Crawler
FirstIndexedBot
FirstIndexedBot is the crawler of FirstIndexed, a directory of newly launched websites. It identifies itself with this user agent:
FirstIndexedBot/1.0 (+https://firstindexed.com/bot)
What it does
- It visits newly registered or newly active domains to see whether a real website has launched.
- On a site, it reads at most a few public pages:
robots.txt, the homepage, up to two contact or about pages, andllms.txt. - It reads only publicly available content. It doesn’t log in, submit forms or run your scripts.
- It makes one request at a time to a site, with short timeouts and a size limit on every page.
It respects robots.txt
FirstIndexedBot follows robots.txt (RFC 9309). To keep it off your whole site:
User-agent: FirstIndexedBot
Disallow: /
Rules for User-agent: * apply too, unless you add a group for FirstIndexedBot.
Questions or removal
To ask a question, report a problem or have your site removed, email hello@firstindexed.com with your domain, or see Remove a listing. Removed sites aren’t listed again.