Glossary term
What is URL frontier? Meaning for developers
The queue of URLs a crawler still has to visit, ordered by priority and by politeness limits per host.
Learn it properly · System Design Case Studies 2Lesson 0: Design a Web CrawlerThe URL frontier, politeness, Bloom-filter dedup and trap defence, with a crawler simulation you can run.