# Crawlie PHP + MySQL backend

This gives you two things, both optional but sharing the same setup:

1. **A real admin panel for the blog** — posts stored in MySQL, visible to every
   visitor, with a proper username/password login (not a shared secret pasted
   into a field).
2. **A much faster, more complete crawl engine** — every tool that crawls a
   whole site (Sitemap Generator, RSS Feed Generator, Google Merchant Feed
   Generator) will use this automatically once it's deployed, instead of
   relying on public CORS relays from the browser.

You don't need to use both — deploying this just for faster crawling, without
ever touching the blog admin, works fine, and vice versa.

## Why this makes crawling faster and stops missing links

Every crawl tool on this site, run purely in the browser, has to fetch other
websites through a public CORS relay (`corsfix.com`, `codetabs.com`, etc.),
because a browser can't fetch a cross-origin page directly without one. Those
relays are free, shared, third-party services — they rate-limit, they
occasionally go down, and a page that fails through all of them shows up as a
"missing" link even though it was really just a relay hiccup, not a dead page.

A PHP script running **on your own server** doesn't have this problem at all.
Server-to-server HTTP requests aren't subject to CORS — there's no relay in
the loop, nothing to rate-limit, nothing with someone else's uptime to depend
on. It can still be blocked by the *target* site's own bot detection (same as
any fetch method, including a real browser), but the entire class of "the
relay itself failed" errors disappears. It's also simply faster: `crawl.php`
fetches dozens of pages at once with real concurrency (`curl_multi`) in a
single request, instead of the browser making one relay round-trip per page.

## 1. Requirements

- PHP 8.0+ with the `curl`, `pdo_mysql`, `dom`, and `xml` extensions (all of
  these ship enabled by default on virtually every shared host)
- A MySQL or MariaDB database
- Apache (recommended) or Nginx with a small config tweak — see `.htaccess`

## 2. Set up the database

```bash
mysql -u youruser -p yourdatabase < schema.sql
```

This creates four tables (`admin_users`, `admin_sessions`, `blog_posts`,
`crawl_jobs`) and seeds one sample blog post so the blog isn't empty on first
load.

## 3. Configure

Edit `config.php`:

```php
define('DB_HOST', '127.0.0.1');
define('DB_NAME', 'yourdatabase');
define('DB_USER', 'youruser');
define('DB_PASS', 'yourpassword');
```

Once your site is live, also tighten `ALLOWED_ORIGIN` from `'*'` to your
actual domain (e.g. `'https://yoursite.com'`), so only your own pages can call
this API from a browser.

## 4. Upload

Upload this whole `backend-php/` folder anywhere under your site's web root,
e.g. `https://yoursite.com/backend-php/`.

## 5. Create your admin login

SSH into your server (or use your host's terminal/cron feature to run a
one-off command) and run:

```bash
php backend-php/create-admin.php yourusername "a strong password"
```

This is CLI-only on purpose — there's no page that can create an admin user
over HTTP.

## 6. Connect the blog admin

Open `blog/admin.html` on your live site. In the "Log in" box, enter:

- **Backend base URL**: `https://yoursite.com/backend-php/api`
- **Username / password**: whatever you set in step 5

That's it — posts you create are now stored in MySQL and visible to every
visitor.

## 7. Connect the faster crawl engine (works independently of the blog login)

Every tool with a "Using your own fetch backend?" section (the audit report,
Sitemap Generator, RSS Feed Generator, Merchant Feed Generator, Competitor
Comparison, and others) has a field for a custom backend URL. Paste in:

```
https://yoursite.com/backend-php/api/fetch.php
```

This alone speeds up every *single-page* fetch across the whole site (no
CORS relay involved). The Sitemap Generator, RSS Feed Generator, and Merchant
Feed Generator go further: they detect that this is `backend-php`'s own
`fetch.php` (by its `/fetch.php` filename) and automatically call its sibling
`crawl.php` for a single fast, complete, server-side crawl instead of dozens
of individual page fetches. Nothing else needs configuring for that — it's
automatic once `fetch.php`'s URL is saved.

## API reference

| Endpoint | Purpose |
|---|---|
| `GET api/fetch.php?url=<target>` | Fetch one page's raw HTML. Drop-in replacement for the Cloudflare Worker option. |
| `GET api/crawl.php?url=<site>&mode=discover` | Full-site URL discovery (sitemap-first, BFS fallback). |
| `GET api/crawl.php?url=<site>&mode=posts` | Same, plus title/description/date extracted per page — feeds the RSS tool. |
| `GET api/crawl.php?url=<site>&mode=products` | Same, but returns only detected products — feeds the Merchant Feed tool. |
| `POST api/login.php` | `{username, password}` returns `{token}` |
| `POST api/logout.php` | Invalidates the current token |
| `GET/POST/PUT/DELETE api/index.php/posts...` | Blog post CRUD (see `api/posts.php` for the exact contract) |

## Limits you can tune

In `config.php`:

- `CRAWL_MAX_PAGES` — hard ceiling on pages per crawl (default 500)
- `CRAWL_CONCURRENCY` — parallel fetches at once (default 12)
- `CRAWL_PER_REQUEST_TIMEOUT` — seconds per individual page (default 12)
- `CRAWL_TOTAL_TIME_BUDGET` — the whole crawl gives up gracefully after this
  many seconds (default 50) — raise this if your host allows longer PHP
  execution times (`max_execution_time`) and you need bigger sites covered
  in one request

## What this still can't fix

A site that renders its content with client-side JavaScript (a React/Vue/
Next.js-style storefront with no server-side rendering) sends an
almost-empty HTML shell to *any* fetch method — a browser via a relay, this
PHP backend, anything that isn't a real headless browser executing the page's
JavaScript. That's outside the scope of a plain HTTP-fetch backend like this
one. The Merchant Feed Generator detects this pattern and tells you plainly
when it happens, rather than just reporting zero products.

A site that actively fingerprints and blocks server IP ranges (aggressive
bot-protection, common on major news sites) may still block this backend the
same way it blocks relays — deploying your own server changes which IP is
making the request, but doesn't make a site's bot-detection stop existing.
