Indeed scraper vs Indeed API: why we deleted ours

We ran an “Indeed scraper” in production for ten months before deleting it for an API. Here’s the breakdown of what actually broke, what it cost, and why the math almost always tips toward managed.

The scraper, technically

Indeed’s public search pages are server-rendered with hydrated React. The job cards are scrapeable from the initial HTML, but the page also fires an XHR for additional metadata and uses Cloudflare-managed bot challenges. To run a production scraper:

• Residential proxy pool (~$500/mo entry, $2k/mo for stable IPs)
• Headless browser fleet for the challenge layer (~$1.5k/mo compute)
• Selector library, updated every 6–8 weeks as Indeed reskins
• Per-region scaling — Indeed serves different HTML to EU vs US

What broke, in order of frequency

Over ten months, our scraper broke for these reasons (most → least):

• Cloudflare challenge changes (5 incidents)
• HTML selector drift after Indeed reskin (3 incidents)
• Per-region rate limit changes (3 incidents)
• Residential IP burning faster than rotation refilled (2 incidents)
• JSON schema change in the hydrated payload (1 incident)

Each incident was 4–12 hours of engineer time. At our blended rate of $180/hr, that’s roughly $1,500–$4,000 per incident, before counting on-call alerting costs and the opportunity cost of the work that didn’t happen because the engineer was firefighting.

The total cost of ownership

Tallied: about $3,800/mo in infra and proxies, plus ~$2,000/mo in engineer maintenance (averaged). That’s $5,800/mo to keep one Indeed scraper alive. JobsPipe’s Growth tier handles the same query volume for $199/mo, with the additional sources thrown in.

We’re biased; you should run your own math. But almost every team that does ends up at the same answer.