A Web Crawler That Doesn't Get Banned
hardOpen designCommonly asked~45 minHLD roundBar raiserSenior / SDE-3GoogleMicrosoftAmazonMetaAdobeSearch & IndexingQueues & AsyncStorage & Data ModellingCapacity & Estimation
The question, as asked
“We need to crawl a few billion pages and keep them reasonably fresh. Design the crawler.”
Scope it before you design it
The prompt is vague on purpose. These are the clarifying questions a strong candidate asks first — answer them out loud before you touch the architecture.
- What's the crawl for — a search index, a price monitor, a training corpus? It decides what you keep and how fresh it has to be.
- Scale and deadline together: a few billion pages in how long? That one division gives you a required pages-per-second, and every later decision follows from it.
- Do we re-crawl, or is this one pass? Recrawl turns a queue into a scheduler with priorities, which is a much more interesting system.
- What are we storing — raw HTML, extracted text, or just links? Raw HTML at a few billion pages is hundreds of terabytes, so this materially changes the storage design.
- Are we rendering JavaScript? A headless browser per page is roughly two orders of magnitude more expensive than an HTTP fetch, so this is a budget question, not a technical one.
- Politeness constraints: are we honouring robots.txt and crawl-delay? Say yes and mean it — this is the constraint that shapes the entire frontier design.
Your answer, first
Commit something before anything unlocks. Saved on this device as you go.
0 charactersExpand for room to draw
Framework hint
Sketch or write your approach above first. The hint gives you the structure, not the answer.