Optimize ListRedisScheduleSource: replace SCAN with sorted set time index

[tmkarthi]

Summary

• Replace the SCAN-based discovery of past time keys with a ZRANGEBYSCORE lookup on a new {prefix}:time_index sorted set, eliminating full keyspace scans that strain Redis when there are many keys
• Track time keys in the sorted set automatically on add_schedule, and clean up stale entries (older than 5 minutes with empty lists) lazily from delete_schedule, rate-limited to once per minute to avoid excess Redis calls
• Fetch past time schedules on every get_schedules call (not just the first run) so that schedules added in a past minute after the previous call are never missed
• Pass current_time from get_schedules into _get_previous_time_schedules to prevent duplicate fetches when the minute boundary crosses between the two calls
• Add populate_time_index constructor parameter for users upgrading from older versions: set to True once to backfill the index via a one-time SCAN, then False for all subsequent runs
• Update README with new Redis key documentation and populate_time_index migration guide

Test plan

• Existing tests for cron, interval, timed schedules, removal, and migration all pass
• test_time_index_populated_on_add — verifies add_schedule writes to the sorted set
• test_time_index_not_eagerly_cleaned_on_delete — verifies no race-prone eager cleanup
• test_cleanup_removes_old_empty_entries — stale empty entries older than 5 min are removed
• test_cleanup_keeps_non_empty_entries — entries with remaining schedules are preserved
• test_cleanup_keeps_recent_empty_entries — entries within 5-min window are not removed
• test_past_schedules_found_via_time_index — past schedules discovered via sorted set
• test_populate_time_index_from_existing_keys — populate_time_index=True backfills index
• test_post_send_triggers_cleanup — full lifecycle: add → get → post_send → cleanup
• test_cleanup_rate_limited — cleanup runs at most once per minute
• test_cron_and_interval_not_in_time_index — non-time schedules don't pollute the index

[tmkarthi]

@s3rius Any update on this?

[s3rius]

First of all. Thanks for you patience. It took me a while to get to your PR.
I really like the solution. It's elegant, simple and yet robust.

I only have a single comment about simplifying migration for people after upgrading their taskiq-redis. However, it's a complete minor thing.

Also, I noticed that you use floats for scores. I'm not sure if turning scores into ints would make a big of a change. But could be, since floats are known to worse at comparison and take a bit more memory than ints. Of course it all depends on how redis implements score type.

[s3rius]

I think that this can be calculated automatically.

if you were using previous version of this schedule source, you are 100% don't have this key set.

We can just check if the key for time_index is present and in case if it's empty, build the initial index. This will allow us to remove the new argument and make a seamless transition.

[Copilot]

Pull request overview

This PR optimizes ListRedisScheduleSource’s handling of time-based schedules by introducing a Redis sorted-set “time index” to avoid keyspace-wide SCANs when discovering past-due time buckets, and adds a migration/backfill option plus documentation and tests.

Changes:

• Add {prefix}:time_index sorted set and switch past schedule discovery from SCAN to ZRANGEBYSCORE.
• Maintain the time index on add_schedule, and lazily clean stale empty entries from delete_schedule with rate limiting.
• Add populate_time_index to backfill the index on startup for upgrades; update README and expand test coverage.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
taskiq_redis/list_schedule_source.py	Implements the time index, backfill-on-startup option, and lazy cleanup; adjusts past schedule fetching behavior.
tests/test_list_schedule_source.py	Adds tests for time index maintenance, lazy cleanup semantics, backfill behavior, and rate limiting.
README.md	Documents the new Redis keys and provides a populate_time_index migration guide.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

_maybe_cleanup_time_index updates _last_cleanup_time before running _cleanup_time_index. If cleanup raises (network hiccup, Redis error), the rate limiter will suppress cleanup retries for 60s even though no cleanup actually happened. Consider only updating _last_cleanup_time after a successful cleanup (or resetting it on exception).

Suggested change

	self._last_cleanup_time = now
	await self._cleanup_time_index(redis)
	await self._cleanup_time_index(redis)
	self._last_cleanup_time = now