The BirdPitch Crawler

BirdPitch is an X (Twitter) outreach tool. Its crawler identifies itself as BirdPitch/1.0 (+https://birdpitch.com/bot).

What it fetches, and when

The crawler fetches a single page: the homepage of a website, at the moment the owner of that website pastes the URL into BirdPitch during onboarding. It reads the title, meta description, headings and visible text to draft a business profile, and it does not crawl further, follow internal links, or return on a schedule.

It is therefore user-initiated rather than continuous. If you see it in your logs, somebody entered your site's address into BirdPitch — in almost every case that is you or a colleague signing up.

Verifying that a request is genuinely from BirdPitch

A User-Agent string can be set by anyone, so we also sign the request. BirdPitch supports Web Bot Auth: signed requests carry Signature-Agent, Signature-Input and Signature headers as described in RFC 9421, using Ed25519.

The Signature-Agent header points at https://birdpitch.com/.well-known/http-message-signatures-directory, a JWKS containing our public key. Fetch it, select the key whose kid matches the keyid in Signature-Input, and verify the signature over the covered components. A request that fails verification is not from us, whatever its User-Agent claims.

Requests made through a headless browser fall back to being identified by User-Agent only, because redirects there are followed internally and a signature minted for the original host would not verify at the redirected one.

Blocking it

The crawler honours robots.txt. To refuse it entirely:

User-agent: BirdPitch
Disallow: /

Blocking has one visible consequence: a customer entering your site into BirdPitch gets a profile drafted from their own description instead of your page's content. Nothing else about the product depends on this crawler.

Contact

Questions about crawler behaviour, or a request to be excluded, can go through the contact route on https://birdpitch.com.