Verified in source: recoverProcessingJobs (src/selfhost/pg-queue.ts ~561-577) does SELECT ... WHERE status='processing' and flips every row to pending with no lease-age check and no AND status='processing' guard on the UPDATE — while the file header advertises multi-instance FOR UPDATE SKIP LOCKED safety.
A new instance booting while the old one is still mid-job (any overlapped or rolling deploy) re-pends the old instance's in-flight jobs. Both then run the same PR pass concurrently — and per #9013 the public-surface publish has no mutex, so this produces duplicate gate check-runs and verdict thrash.
Related, same file: reclaimExpiredProcessingJobs (~1609-1629) skips only ids in this instance's in-memory activeJobIds, so a genuinely long (>30 min) pass can be reclaimed and double-run by a sibling. It also never increments attempts, so a job that reliably wedges past the lease loops forever without ever dead-lettering.
Fix
- At boot, recover only rows older than the lease (reuse
reclaimExpiredProcessingJobs's cutoff), or skip boot recovery on Postgres entirely and rely on the runtime reaper.
- Heartbeat the lease from the worker and increment
attempts on reclaim so chronic wedgers dead-letter.
Refs #9007, #9013.
Verified in source:
recoverProcessingJobs(src/selfhost/pg-queue.ts~561-577) doesSELECT ... WHERE status='processing'and flips every row topendingwith no lease-age check and noAND status='processing'guard on the UPDATE — while the file header advertises multi-instanceFOR UPDATE SKIP LOCKEDsafety.A new instance booting while the old one is still mid-job (any overlapped or rolling deploy) re-pends the old instance's in-flight jobs. Both then run the same PR pass concurrently — and per #9013 the public-surface publish has no mutex, so this produces duplicate gate check-runs and verdict thrash.
Related, same file:
reclaimExpiredProcessingJobs(~1609-1629) skips only ids in this instance's in-memoryactiveJobIds, so a genuinely long (>30 min) pass can be reclaimed and double-run by a sibling. It also never incrementsattempts, so a job that reliably wedges past the lease loops forever without ever dead-lettering.Fix
reclaimExpiredProcessingJobs's cutoff), or skip boot recovery on Postgres entirely and rely on the runtime reaper.attemptson reclaim so chronic wedgers dead-letter.Refs #9007, #9013.