ai-robots-txt

ai-robots-txt/ai.robots.txt

A list of AI agents and robots to block.

4.0K stars 168 forks 4.0K watchers Python MIT License

ai crawlers crawling privacy

View on GitHub Website

15 Open Issues Need Help Last updated: Jul 9, 2026

Open Issues Need Help

View All on GitHub

releases 1.48++ are broken (anchoring for nearly all regexps) about 9 hours ago

bug help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Missing ExaSearchBot 12 days ago

enhancement good first issue help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

nbot traffic 14 days ago

enhancement good first issue help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Matching a string of "spider" catches too much 22 days ago

bug help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

nginx expressions escapes too much and matches case-insensitive 23 days ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Blocking Mozilla Tabstack 27 days ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Blocks GitLab Visual Studio Code extension about 2 months ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

robots.txt falling behind robots.json 3 months ago

bug help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Bingbot Quandry 4 months ago

documentation help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

new user-agent 5 months ago

enhancement help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

False positive on old Internet Explorer / Trident based browsers containing SLCC in user agent string 7 months ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

GrokAppAndroid 8 months ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Test dark visitors update and improve code/test_files/robots.json 8 months ago

good first issue help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

python requirements 8 months ago

good first issue help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Block IbouBot 10 months ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy