# NOTE: Cloudflare prepends a "Managed content" block (Content-Signal + AI
# crawler rules) to this file at the edge. The live rentremote.com/robots.txt is
# therefore larger than this file and contains a second "User-agent: *" group
# with "Allow: /". Google merges same-agent groups and resolves by longest match,
# so the Disallow rules below still apply. That block is configured in the
# Cloudflare dashboard, not here.
#
# Every Disallow MUST start with "/". A pattern beginning with "*" matches that
# substring anywhere in the URL, not as a path prefix. This file previously used
# "Disallow: *location*", which blocked every listing slug ending in
# "-in-central-location" (56K canonical pages that we publish in our own
# sitemap). tests/Unit/Seo/RobotsTxtTest.php enforces the anchoring.

User-agent: *

# Authentication is NOT disallowed. /login and /register both serve
# "noindex, nofollow" (App\Support\AuthPageSeo), and Googlebot has to be able
# to fetch them to see it. Blocking /login here hid that directive: the
# /login?redirect={article} URLs minted before #1100 could never be dropped,
# they just sat in Search Console's "Blocked by robots.txt" bucket, and every
# validation run against them failed by construction because the block was
# working as written. The discovery chain is already cut on /register, which is
# crawlable and carries rel="nofollow" on its link to /login.
# tests/Unit/Seo/RobotsTxtTest.php enforces that /login stays fetchable.

# The private booking funnel is NOT disallowed either, because those paths are
# unreachable for a crawler rather than merely unlinked:
#
#   /properties/123            404      /reservations/456          404
#   /reservation/456           404      /reservation-payments/1    404
#   /l/apartments/x/customize  404      /properties                301 -> /a/furnished-apartments
#
# The reservation rows above are 404 by enforcement, not by accident. A guest
# reservation is addressed by its 7-character booking code, and the
# 'reservation.booking-code' middleware refuses every other way of addressing
# it, including the numeric-id fallback in Reservation::resolveRouteBinding().
# A crawler can only ever synthesise a numeric id, so it can only ever get a
# 404. It cannot reach the booking-code form: those URLs are unguessable and
# appear only in mail sent to the guest.
#
# This is load-bearing, so do not treat it as a comment. Until 2026-09-03 the
# numeric form of these paths returned 200 with the guest's real name and email
# rendered in the page, which made this paragraph false at exactly the moment
# the Disallow rules were removed.
# tests/Feature/Security/UnauthenticatedReservationAccessTest.php enforces the
# 404 on every funnel path listed here. If that test is ever weakened, the
# Disallow rules have to come back with it.
#
# A 404 removes a URL from the index on its own. Blocking it means Googlebot
# can never fetch it to see the 404, so the URL is never dropped: it just
# accumulates in Search Console's "Blocked by robots.txt" bucket, which stood
# at 34,780 pages on 2026-08-28.
#
# /locations and /coworkings are the exception that proves the rule. They
# return 200 and were held out of the index by this file alone, so they now
# serve "noindex, follow" of their own (LocationsPageController and
# layouts/base.blade.php via @yield('robots')). Removing the block is what lets
# Googlebot read that. /locations/{city} was already showing up as "Indexed,
# though blocked by robots.txt" precisely because the 404 it returns was hidden.
#
# tests/Unit/Seo/RobotsTxtTest.php enforces that the noindex-only paths stay
# fetchable; tests/Feature/Seo/InternalIndexPageIndexingTest.php enforces the
# noindex.

# Date-filtered listing URLs (?checkin=&checkout=) are intentionally NOT blocked
# here. PropertyController::seoData and FurnishedApartmentsController::
# listingRobotsPolicy already emit "noindex, follow" for any URL with a query
# string, and Googlebot has to be able to fetch the page to see that tag. A
# robots.txt block would hide the noindex and strand those URLs in the index.

# Spam paths (/shop/, /product/, /goods/, *.phtml) from the 2024 keyword hack
# are intentionally NOT disallowed here: BlockChineseSpamUrls middleware serves
# them a 410 Gone, and Googlebot must be able to crawl them to see the 410 and
# drop the ~89K stale URLs from its index. A robots.txt block keeps them stuck
# in Search Console's "Blocked by robots.txt" bucket forever.

Sitemap: https://rentremote.com/sitemap.xml