fix(scrapy): stop the crawl gracefully when the Actor run is migrated or aborted

[vdusek]

Closes #1103.

The Scrapy integration ignored the MIGRATING and ABORTING events, so the process was killed with requests in flight. Those stay pending and get downloaded again by the next run, and every item their callbacks had already pushed lands in the dataset twice.

The issue proposed crawler.stop_async() for both events, but that is wrong for a migration: the platform finishes a run whose container exits with code 0 as SUCCEEDED and doesn't restart it (apify-worker act2_run_job.ts, only a non-zero exit with wasMigrationInitiated re-schedules the run). A crawl that stopped on MIGRATING would end the run as successful with work still pending. A graceful abort is different: the run is already DONE when aborting is emitted, so exiting with 0 yields ABORTED.

Migration

ApifyScheduler listens for MIGRATING. From then on next_request() hands out nothing, and the requests Scrapy is still working on are marked as handled as they finish. The crawl stays alive and idle until the platform kills it, so the next run continues with the pending requests instead of downloading the finished ones again. Actor.reboot() dispatches the same listener and awaits it, so a reboot now waits for the in-flight requests to settle first.

Abort

The new ApifyGracefulStopExtension, registered by apply_apify_settings, calls crawler.stop_async() on ABORTING: the in-flight requests finish, the scheduler marks them as handled, and the run ends as ABORTED within the grace period.

Docs and tests

The migration section of the Scrapy guide describes the behavior and the start-URL caveat: the default Spider.start() yields with dont_filter=True (mapped to always_enqueue=True), so a restarted run crawls the start URLs again. The example spider overrides start() with plain requests. As a side effect the start page now deduplicates against the same link found on subpages, so the title-spider e2e test asserts unique items instead of a count that included that duplicate.

Two e2e tests exercise the paths on the platform: a graceful abort with a request in flight (run ABORTED, spider closed gracefully, dataset items match the handled requests, the rest of the chain stays pending) and an Actor.reboot() mid-crawl (the in-flight request is settled before the reboot, the rebooted run finishes the chain with no duplicates).

✍️ Drafted by Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.82%. Comparing base (5a0a752) to head (8cc1e4c).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1104      +/-   ##
==========================================
+ Coverage   92.65%   92.82%   +0.16%     
==========================================
  Files          51       52       +1     
  Lines        3445     3510      +65     
==========================================
+ Hits         3192     3258      +66     
+ Misses        253      252       -1     

Flag	Coverage Δ
e2e	?
integration	53.47% <0.00%> (-1.01%)	⬇️
unit	89.40% <100.00%> (+0.22%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.