FirstIndexed Is my site listed?
Crawler

FirstIndexedBot

FirstIndexedBot is the crawler of FirstIndexed, a directory of newly launched websites. It identifies itself with this user agent:

FirstIndexedBot/1.0 (+https://firstindexed.com/bot)

What it does

  • It visits newly registered or newly active domains to see whether a real website has launched.
  • On a site, it reads at most a few public pages: robots.txt, the homepage, up to two contact or about pages, and llms.txt.
  • It reads only publicly available content. It doesn’t log in, submit forms or run your scripts.
  • It makes one request at a time to a site, with short timeouts and a size limit on every page.

It respects robots.txt

FirstIndexedBot follows robots.txt (RFC 9309). To keep it off your whole site:

User-agent: FirstIndexedBot
Disallow: /

Rules for User-agent: * apply too, unless you add a group for FirstIndexedBot.

Questions or removal

To ask a question, report a problem or have your site removed, email hello@firstindexed.com with your domain, or see Remove a listing. Removed sites aren’t listed again.