Glossary term
What is politeness? Meaning for developers
Limits that stop a crawler from overloading one website: robots.txt rules and a delay between requests to the same host.
Learn it properly · System Design Case Studies 2Lesson 0: Design a Web CrawlerThe URL frontier, politeness, Bloom-filter dedup and trap defence, with a crawler simulation you can run.