# postallads4free.com — the reasoning lives in docs/seo-indexing.md
#
# Nothing is disallowed for the general crawler, deliberately. The one
# exception is Yandex, blocked outright at the bottom of this file.
#
# The crawl path we need walked is:
#   footer (every page) -> /browse -> /search?loc_id=..&cat_id=.. -> /ad{id}.html
# A clean city+category hub with enough listings emits index,follow — that
# changed on 2026-08-22 when the hubs were opened, and this comment said
# noindex for three days after it. /browse and every filtered, sorted, free-text
# or paginated variant is still noindex,follow: crawlers are meant to walk
# through those, and `page=` is how they get past the first 20 ads on a hub.
#
# The hub filter parameters (searchtext, order, ad_type, item_state, beds,
# baths) were briefly disallowed here on crawl-budget grounds and reverted the
# same day. Three reasons, worth keeping so it is not re-tried blind:
#
#   1. Those filtered variants are noindex. Blocking them means Google never reads
#      the tag, so any variant that is already indexed or externally linked
#      gets stuck as a URL-only entry instead of dropping out cleanly.
#   2. Blocking the URL blocks the `follow` with it, cutting links into ad
#      pages that we spent a release building.
#   3. `Disallow: /search?*order=` matches anything containing "order=" — a
#      future parameter, or a per-category dynamic filter whose attribute name
#      ends that way. Silent over-blocking.
#
# The filters are form-driven rather than linked, so there is little for a
# crawler to find in the first place. If crawl waste on this parameter space
# ever needs fixing, MEASURE IT FIRST — Search Console > Settings > Crawl
# stats, or the nginx access log — and block only what is demonstrably being
# requested.

# Yandex is barred from the site entirely, and it is the one exception to the
# "nothing is disallowed" rule above.
#
# GEOGRAPHY_EXCLUDED_COUNTRY_CODES (by,cu,ir,kp,ru,sy,ve) is an application
# policy list: Country::scopeAllowed() and Locality::scopeAllowed() keep those
# countries out of the gazetteer download, the import, and every selector and
# search query, so there is no RU locality to post an ad against. Yandex indexes
# for the one market this configuration exists to refuse, so its crawl buys
# nothing and costs bandwidth on 20k ad pages.
#
# `User-agent: Yandex` is the token that covers all their robots (YandexBot,
# YandexImages, ...), not just the main one.
#
# NOTE this does not stop IndexNow: that protocol forwards every submitted URL
# to every participant, Yandex included, with no opt-out — see
# docs/seo-indexing.md. Blocking the crawler and then pinging the protocol would
# hand Yandex a URL list it is simultaneously forbidden to fetch.
User-agent: Yandex
Disallow: /

User-agent: *
Disallow:

Sitemap: https://postallads4free.com/sitemap.xml