You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Disable automatic redelivery for long-running RedisStreamBroker tasks? #101
We have a cron-like task that scans a large Redis dataset and takes ~35 minutes to complete.
With RedisStreamBroker, the default idle_timeout is 600_000 ms (10 minutes).
Once the worker runs longer than that, TaskIQ calls XAUTOCLAIM, redelivers the message, and a second instance starts (ours cancels itself via a custom lock, but it still creates noise).
Right now, the only workarounds we see are:
Split the long task into shorter pieces.
Set idle_timeout to a huge value (we currently use one year in milliseconds).
Questions:
Is there any supported way to completely disable automatic redelivery, so a job is never re-queued while its worker is alive?
If disabling isn’t possible, what’s the recommended approach for genuinely long-running jobs?
Was the fixed redelivery behavior chosen mainly for resilience against crashed workers, or do you expect users to handle long tasks differently?
Instead of reclaiming solely from a fixed PEL idle timeout, it uses a TTL-backed heartbeat per worker consumer. A long-running task is not reclaimed while its worker is alive; when a worker crashes, its heartbeat expires and another worker can reclaim its pending entries.
It is an independent broker rather than a taskiq-redis patch, but this heartbeat-based reclaim policy may be a useful direction for this issue.
Available on PyPI as taskiq-redis-streams (0.2.0).
Activity
adiberk commentedon Dec 23, 2025
I believe you can disable via labels - and maybe add custom code to the broker to read that label and take that into consideration
vvanglro commentedon Aug 4, 2026
I built taskiq-redis-streams to address this case.
Instead of reclaiming solely from a fixed PEL idle timeout, it uses a TTL-backed heartbeat per worker consumer. A long-running task is not reclaimed while its worker is alive; when a worker crashes, its heartbeat expires and another worker can reclaim its pending entries.
It is an independent broker rather than a
taskiq-redispatch, but this heartbeat-based reclaim policy may be a useful direction for this issue.Available on PyPI as
taskiq-redis-streams(0.2.0).