Skip to content

Disable automatic redelivery for long-running RedisStreamBroker tasks? #101

Open

Description

@smalyu

We have a cron-like task that scans a large Redis dataset and takes ~35 minutes to complete.
With RedisStreamBroker, the default idle_timeout is 600_000 ms (10 minutes).
Once the worker runs longer than that, TaskIQ calls XAUTOCLAIM, redelivers the message, and a second instance starts (ours cancels itself via a custom lock, but it still creates noise).

Right now, the only workarounds we see are:

  1. Split the long task into shorter pieces.
  2. Set idle_timeout to a huge value (we currently use one year in milliseconds).

Questions:

  • Is there any supported way to completely disable automatic redelivery, so a job is never re-queued while its worker is alive?
  • If disabling isn’t possible, what’s the recommended approach for genuinely long-running jobs?
  • Was the fixed redelivery behavior chosen mainly for resilience against crashed workers, or do you expect users to handle long tasks differently?

Thanks!

Activity

  1. adiberk commented on Dec 23, 2025

    @adiberk

    I believe you can disable via labels - and maybe add custom code to the broker to read that label and take that into consideration

  2. vvanglro commented on Aug 4, 2026

    @vvanglro

    I built taskiq-redis-streams to address this case.

    Instead of reclaiming solely from a fixed PEL idle timeout, it uses a TTL-backed heartbeat per worker consumer. A long-running task is not reclaimed while its worker is alive; when a worker crashes, its heartbeat expires and another worker can reclaim its pending entries.

    It is an independent broker rather than a taskiq-redis patch, but this heartbeat-based reclaim policy may be a useful direction for this issue.

    Available on PyPI as taskiq-redis-streams (0.2.0).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions