Flowpane Web Crawler

About

Flowpane is a web governance platform that helps digital agencies monitor compliance files across their client websites. Our crawler periodically fetches governance files to verify they exist, are correctly configured, and follow best practices.

We do not crawl page content, collect personal data, or index websites for search. Our checks are limited to publicly accessible governance and compliance files only.

Identification

User-Agent	Flowpane/1.0 (+https://crawler.flowpane.com)
Compatible UA	Mozilla/5.0 (compatible; Flowpane/1.0; +https://crawler.flowpane.com)
Operator	Flowpane Ltd
Contact	crawler@flowpane.com
Website	flowpane.com

Behaviour

Files accessed — robots.txt, ads.txt, security.txt, llms.txt, humans.txt, sitemap.xml, sitemap_index.xml, and cookie/consent detection on the homepage
Request rate — Maximum 1 request per second per domain. Respects Crawl-delay directives in robots.txt
robots.txt compliance — Fully RFC 9309 compliant. Respects Disallow rules for both Flowpane and * user-agent groups
Redirects — Follows up to 5 redirects (301, 302, 307, 308)
Timeout — 3-second connection timeout per request
No JavaScript — Raw HTTP responses only. No headless browser execution during automated checks
Caching — Results cached per site's check schedule. Minimum 24 hours between automated checks for any given file
Request signing — All requests are cryptographically signed using Web Bot Auth (HTTP Message Signatures with Ed25519). Cloudflare Verified Bot applicant.

IP Addresses

Flowpane's crawler operates from the following IP address. You can use this to create WAF allowlist rules.

46.32.253.56

Reverse DNS: 46.32.253.56 resolves to crawler.flowpane.com. This list will be updated if additional IPs are added.

Verifying Requests

Flowpane uses three layers of verification so site owners and WAF operators can confirm a request is genuinely from us.

1
User-Agent header
Check that the User-Agent contains Flowpane/1.0 and references this page.

Primary: Flowpane/1.0 (+https://crawler.flowpane.com)
2
IP address and reverse DNS
Verify the source IP is in the published range above, and that a reverse DNS lookup resolves to crawler.flowpane.com.

Forward: crawler.flowpane.com → 46.32.253.56
Reverse: 46.32.253.56 → crawler.flowpane.com
3
Cryptographic request signing (Web Bot Auth) Ed25519
Every request from Flowpane carries a cryptographic signature using HTTP Message Signatures (Web Bot Auth). This proves the request originates from Flowpane and hasn't been tampered with.

Signed components: @authority, signature-agent
Algorithm: Ed25519
Signature validity: 5 minutes
Public key directory: /.well-known/http-message-signatures-directory

Verifying a signature

Incoming requests from Flowpane include three headers:

Signature: sig=:<base64-encoded-signature>:
Signature-Input: sig=("@authority" "signature-agent");created=...;expires=...;keyid="...";alg="ed25519";tag="web-bot-auth"
Signature-Agent: crawler.flowpane.com

To verify: fetch the public key from the key directory, extract the key matching the keyid parameter, and verify the Ed25519 signature over the signed components. Libraries like web-bot-auth (npm) handle this automatically.

Requests claiming to be from Flowpane that fail any of these three checks are spoofed. Report them to crawler@flowpane.com.

Whitelisting

If your WAF is blocking Flowpane, use the instructions below for your provider.

Flowpane has applied to be a Cloudflare Verified Bot with cryptographic request signing (Web Bot Auth). Once approved, Flowpane will be automatically recognised and allowed by Cloudflare's bot management without any action needed from you.

In the meantime, you can whitelist manually:

Navigate to Security → WAF → Custom Rules
Create a rule with condition: User-Agent contains Flowpane/1.0
Set the action to Allow
Alternatively, add Flowpane's IP to Security → IP Access Rules

In Bot Manager, navigate to your bot policy
Add Flowpane/1.0 to the allowed bot list
Or create an IP whitelist for 46.32.253.56

Log in to the Sucuri dashboard
Go to Settings → Security → Whitelist
Add the Flowpane user agent or IP address

Add a firewall rule to allow requests where the User-Agent header contains Flowpane/1.0
Or whitelist IP address 46.32.253.56
Verify via reverse DNS: dig -x 46.32.253.56 should return crawler.flowpane.com

Opting Out

If you don't want Flowpane to access your site, add the following to your robots.txt:

User-agent: Flowpane
Disallow: /

Our crawler fully respects robots.txt directives. This will prevent all automated Flowpane checks.

You can also contact us at crawler@flowpane.com and we'll exclude your domain manually.