# Optional fetch backend (fixes sites that block every public proxy)

Crawlie normally fetches live URLs entirely from your browser, by routing
through public CORS relays. That works for most sites, but some — especially
news sites and anything behind aggressive bot protection — block those relays
outright, because their IP ranges and User-Agent are well known.

There is no way to fix this from client-side JavaScript alone: the block
happens on the target site's server, before your browser's code ever runs.
The only real fix is a small server-side proxy you control, which can use a
normal browser User-Agent and isn't on anyone's public blocklist (yet).

This folder gives you two ready-to-deploy options that do exactly that.
Pick **one**.

## Option A — Cloudflare Worker (recommended: free, no build step, 2 minutes)

1. Go to https://dash.cloudflare.com → **Workers & Pages** → **Create** → **Worker**.
2. Delete the sample code in the editor and paste in the entire contents of
   `cloudflare-worker/worker.js`.
3. Click **Deploy**. You'll get a URL like `https://crawlie-fetch.<you>.workers.dev`.
4. Open the audit report page on your Crawlie site, run a scan that fails,
   and in the "Couldn't fetch automatically" panel expand
   **"Using your own fetch backend?"**, paste that URL in, and click **Save**.
5. Click **Try fetching again**.

Free tier: 100,000 requests/day — far more than a personal tool needs.

## Option B — Vercel serverless function

1. Create (or reuse) a Vercel project.
2. Add `vercel-function/api/fetch.js` to that project as `api/fetch.js`.
3. Deploy. Your endpoint is `https://<your-project>.vercel.app/api/fetch`.
4. Same as step 4–5 above.

## Option C — PHP + MySQL (if you're on ordinary shared/LAMP hosting)

See `../backend-php/README.md`. This one does more than the other two: on
top of the same single-page fetch fix, it adds a fast, complete server-side
crawl engine (used automatically by the Sitemap Generator, RSS Feed
Generator, and Merchant Feed Generator once configured) and a real MySQL-backed
admin login for the blog. Worth it specifically if you already have PHP+MySQL
hosting and want the blog's admin panel to be a real shared backend rather
than local-only, or if you're regularly crawling large sites and want that to
be both faster and miss fewer pages than the public-relay approach.

## Honest limits

Both options do a plain server-side HTTP fetch with a real browser
User-Agent — they are **not** a headless browser. Sites using heavy
JavaScript-challenge bot protection (Cloudflare "I'm Under Attack" mode, full
browser fingerprinting/TLS fingerprinting, login walls, etc.) can still block
even this. Nothing short of running an actual headless browser (e.g.
Playwright) on a server can guarantee 100% of sites are fetchable — and even
that isn't truly guaranteed against determined anti-bot systems.

What this *does* reliably fix: the large majority of sites that were only
blocking the public relays because of their generic headers or blocklisted
IPs — which is most of them.

If a site still can't be fetched after setting this up, the **paste HTML
source** option on the report page always works, because it doesn't depend
on any network fetch at all.
