What Israeli local authorities' robots.txt files tell AI crawlers

The counts

Local authorities on the list

259

With a site address in official lists

202

Returned a robots.txt file

75

Of those, block at least one answer crawler

0

202 of the 259 authorities have a site address in the government lists we used. Our fetch got a robots.txt file back from 75 of 202 (37.1%). Of the rest, 66 answered with a refusal or not found (4xx), and 3 sent an HTML page. For 25, the old address now redirects to another domain, so we did not count that file as the authority's. 1 returned a server error, and 32 gave no answer at all.

Our requests came from a home internet connection in Israel, and 1 of the refusals was 403 Forbidden. A first run the same day from a data centre in Germany got 96 answers of 403. Many of these sites refuse requests from abroad or from data centres, which is where most crawlers run. The method section has the details.

None of the 75 files blocks any of the 11 answer crawlers from the home page. 12 block at least one of GPTBot, ClaudeBot and CCBot. 43 name a sitemap.

Every authority

Names are as the government lists give them, in English where the Central Bureau of Statistics has one and in Hebrew always. Sort by any column, or type to filter. Without JavaScript the table is in alphabetical order and complete.

"Blocked" means the file's rules for that crawler disallow the home page, /. A crawler that a file does not name follows its * group, as RFC 9309 says. The CSV also lists the crawlers each file names, and Google-Extended.

How we built the list and fetched each file

The authorities

The list is the National Digital Agency's consolidated list of municipal authorities on data.gov.il. It was last updated on 10 December 2025 and has 259 authorities, each with its type and district. For cities and local councils, the English name, map position and district come from the Central Bureau of Statistics localities file for 2023. The two files share the authority code. For four authorities the list gives a different district, or none, and we follow the statistics file. It has no rows for regional councils as such. Their district is the list's, their name is in Hebrew only, and the map puts each one at the average position of its localities.

The site addresses

Neither file lists websites, so we took each address from government lists only. The first choice is the website field of the Ministry of Culture and Sport's list of local authority sports departments (2019). Failing that, we used the domain of the authority's own email address in official contact lists. The Ministry of Interior's list of heads of local authorities (2019) supplied 141 of them. The Ministry of Welfare's volunteering coordinators (2025) supplied 5. We also read the Ministry of Aliyah and Integration's local coordinators (October 2026) and the Ministry of Interior's election managers (2025). They confirmed some addresses and added none.

Among email domains, a muni.il domain wins, because Israel's domain registry gives muni.il names to local authorities only (ISOC-IL registration rules). Next comes a domain in the heads of authorities list, then the domain the most lists agree on. Public mail services, umbrella bodies and domains shared by several authorities are left out. Where none of this gave an address, the row says "No site address found". That means the lists we used don't name one. It doesn't mean the authority has no site. The CSV says where each address came from.

The fetch

One request for https://<site>/robots.txt per site, on 9 Nov 2026, at most two requests a second in total, counting every redirect. Redirects were followed up to five hops. Each request waited up to 15 seconds. Where a www. name did not resolve, we used the bare domain. The user agent named us and the tool:

SynapseRobotsSurvey/1.0 (one robots.txt fetch per site; +https://github.com/synapsereality/answer-crawler-check; https://synapsereality.io)

The requests came from a home internet connection in Israel. Some sites answer requests from outside Israel differently, or not at all. A crawler that fetches from somewhere else, or with its own user agent, can get a different answer from the one in this table.

How a file is read

Each file was parsed with the library inside answer-crawler-check 0.1.1, our open-source robots.txt checker. It picks a crawler's group and the winning rule the way RFC 9309 and Google describe. The answer crawlers are its default list: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Applebot, Googlebot, Bingbot. Googlebot and Bingbot are on it because Google's AI features and Copilot answer from those indexes. The training crawlers are GPTBot, ClaudeBot, CCBot. Google-Extended is a separate token. Google's page says it controls training of Gemini models and grounding in Gemini Apps and Vertex AI (our token table quotes it).

RFC 9309 also says what a crawler does when there is no file. On a 4xx it may crawl everything. On a 5xx or no answer it should assume it may crawl nothing. The table reports the status we got and leaves that reading to you. A 2xx response sent as HTML, by its content type or its first characters, was not parsed. Neither was a file reached by a redirect to a different domain, unless that domain is a muni.il name.

National context

Zyte's State of Web Access 2026 profile of Israel looked at 1,109 Israeli sites of every kind. It reports that 11.8% of them disallow GPTBot, 8.7% CCBot and 8.5% ClaudeBot. On 15.8% of them, robots.txt names AI crawlers. Those figures are Zyte's, from a different sample and method. We read them on 9 November 2026. Nothing on this page is drawn from them.

Corrections

If a site address here is wrong, or an authority's site is missing, tell us with the official page that lists it. We'll fetch that site once, update the row and date the change.