Scrapy: stop the crawl gracefully on migration and abort to avoid duplicate items

[vdusek]

Problem

The Scrapy integration does not react to the MIGRATING and ABORTING platform events, so the process is killed with requests in flight. Since #1097 those requests stay unhandled in the request queue and the next run downloads them again, which is correct, but every item their callbacks had already pushed lands in the dataset a second time. The Scrapy guide lists this as a known limitation with the HTTPCACHE_* workaround (apify/actor-templates#303); that only makes the re-download cheap, it does not prevent the duplicates.

Crawlee handles exactly this case: on MIGRATING / ABORTING it pauses the pool for up to 20 s (SAFE_MIGRATION_WAIT_MILLIS) so that in-flight requests finish and get marked as handled.

Proposal

1. Add a Scrapy extension, registered by apply_apify_settings, that subscribes to Event.MIGRATING and Event.ABORTING on spider_opened and calls await crawler.stop_async() (Scrapy >= 2.14). The engine stops scheduling new requests, waits for the in-flight ones including their callbacks and item pipelines, and only then closes the scheduler, whose close() marks everything finished as handled. Only callbacks still running at the hard kill can still produce duplicates.
2. Document that the default Spider.start() yields the start URLs with dont_filter=True, which the integration maps to always_enqueue=True, so the start URLs are crawled again after every restart. Overriding start() with plain Requests makes them deduplicate against the request queue.
3. Rewrite the "Dealing with imminent migration to another host" section of the Scrapy guide accordingly.

✍️ Drafted by Claude Code