Skip to content

fix(invalidation): throttle the listener's reconnect WARNING to one a minute (LAB-7638) - #482

Merged
27Bslash6 merged 1 commit into
mainfrom
fix/LAB-7638-listener-warning-throttle
Oct 3, 2026
Merged

27Bslash6 merged 1 commit into
mainfrom
fix/LAB-7638-listener-warning-throttle

Conversation

@27Bslash6

Copy link
Copy Markdown
Contributor

What

While Redis is unreachable or rejects AUTH, the invalidation listener fails once a second, and each failure logged a WARNING Invalidation listener error; retrying in 1 s. A long outage across many workers made that a log flood. The branch now goes through a _WarnThrottle, as the module's publish and dropped-event failures already do: one WARNING a minute per process, Invalidation listener error (errors since the last warning: N), with DEBUG in between. The 1 s retry and the ResponseError (refused subscription) branch are unchanged.

Also in this PR

  • docs/features/l1-invalidation.md: the Delivery bullet and the troubleshooting entry describe the throttled WARNING; the "No WARNING" list adds the silently dead connection case the Delivery bullet already documents.
  • docs/configuration.md: lists CACHEKIT_INVALIDATION_LISTENER_ENABLED with a link to Cross-Process L1 Eviction.
  • tests/integration/test_invalidation_channel_redis.py: the ACL tests close the restricted client in finally, so a failing assertion no longer leaves it open.
  • tests/unit/test_invalidation_listener.py: new test_connection_errors_warn_once_a_window_with_the_count (fails with the throttle removed). Removed three tests whose coverage lives elsewhere: TestSWRRefreshAfterEviction (within-TTL SWR runs only in L1-only mode, where no listener registers; test_object_cache.py covers the guard), test_flag_reads_the_environment, and the py-call-osafterfork case of test_import_warns_once (covered by test_any_fork_option_silences_it).

Testing

ruff check, ruff format --check, basedpyright, pytest tests/unit tests/critical -m "not slow" (4571 passed), tests/integration/test_invalidation_channel_redis.py against a live Redis (18 passed), doctests, tests/docs/ and --markdown-docs README.md docs/.

Closes LAB-7638

… minute (LAB-7638)

While Redis is unreachable the listener fails once a second, and each
failure logged a WARNING. Route it through a _WarnThrottle like the
module's other background failures: one WARNING a minute per process with
the count since the last one, DEBUG in between. The 1 s retry is unchanged.

Also: document the throttled WARNING and the silently dead connection case,
list CACHEKIT_INVALIDATION_LISTENER_ENABLED in configuration.md, close the
restricted client in the ACL tests' finally blocks, and drop three tests
that duplicate coverage elsewhere.
@coderabbitai

coderabbitai Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: cachekit-io/cachekit-py/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 3de27981-4e57-4e3d-abe4-525fc1c95092
📥 Commits

Reviewing files that changed from the base of the PR and between 72026b1 and 600587f.

📒 Files selected for processing (5)
  • docs/configuration.md
  • docs/features/l1-invalidation.md
  • src/cachekit/invalidation.py
  • tests/integration/test_invalidation_channel_redis.py
  • tests/unit/test_invalidation_listener.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


Summary by CodeRabbit

  • Bug Fixes
    • Connection-error warnings from the cross-process cache invalidation listener are now limited to one per minute and report how many errors occurred since the previous warning. Other errors are logged at DEBUG level; reconnection attempts continue every second.
  • Documentation
    • Added documentation for enabling the invalidation listener on the tenant-scoped Redis backend, including its default setting and per-process behaviour.
    • Expanded troubleshooting guidance on connection failures and the possibility of stale L1 cache entries until their time-to-live expires.

Walkthrough

The invalidation listener now rate-limits connection-error warnings and includes the error count in each warning. It logs intervening errors at DEBUG and continues retrying every second. Documentation and tests describe or check the warning behaviour, and ACL integration tests close restricted clients only when creation succeeds.

Changes

Invalidation listener updates

Layer / File(s) Summary
Throttle listener error warnings
src/cachekit/invalidation.py, tests/unit/test_invalidation_listener.py, docs/configuration.md, docs/features/l1-invalidation.md
The listener emits counted connection-error warnings at most once per minute. It logs intervening errors at DEBUG and retains the one-second retry. Unit tests cover the warning behaviour. Documentation describes the listener setting, warning count, and retry behaviour.
Update listener test cleanup
tests/integration/test_invalidation_channel_redis.py, tests/unit/test_invalidation_listener.py
ACL integration tests close restricted clients only if creation succeeds. Unit tests remove an environment-value check and simplify the uWSGI warning assertion.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 60058

The change bounds repeated warning noise without changing the retry cadence, and the supplied evidence identifies no issue that needs resolution before merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 3 files. (2 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the listener warning throttle and its one-minute interval.
Description check ✅ Passed The description explains the problem, the changes, and the reported tests. It does not use the template headings or complete the type-of-change and other checklist items, but it provides enough inform…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 3 files. (2 skipped: 2 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kodus-27b

kodus-27b Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Code Review Completed! 🔥

The code review was successfully completed based on your current configurations.

Kody Guide: Usage and Configuration
Interacting with Kody
  • Request a Review: Ask Kody to review your PR manually by adding a comment with the `@kody start-review` command at the root of your PR.

  • Provide Feedback: Help Kody learn and improve by reacting to its comments with a 👍 for helpful suggestions or a 👎 if improvements are needed.

Providing Context (Files & MCPs)

Add these hints in your PR description (or a comment) to unlock deeper checks:

  • Ticket / Acceptance Criteria: `Refs: ABC-123` (Linear/Jira/Asana/ClickUp/Trello) or a direct ticket link.
  • Bugfix Validation: a Sentry/Datadog/Bugsnag event link (or paste the stack trace/error message).
  • Endpoint Risk: mention the route (e.g., `POST /api/payments`) or controller/action name.
  • Attach a repo file as context: use an explicit marker like `@file:docs/guide.mdx#L10-L50` (replace with your real path).
  • API Contract Docs: include `@file:openapi.yaml` or `@file:swagger.json` when changing routes/schemas.
  • Definition of Done / Standards: include `@file:DOD.md` or `@file:CONTRIBUTING.md` if your repo has them.
  • Design System Source of Truth: include `@file:ui/index.ts` (replace with your DS entrypoint path).
  • Feature Flags: include the flag key/name and `@file:flags.ts` / `@file:config.json` (and optionally the PostHog flag name).
  • Edge/CDN Rules: link the Cloudflare rule/zone or describe the intended redirect/header behavior.
  • Attach an MCP tool output: use `@mcp<provider|tool>` (replace with an installed MCP provider + tool, e.g., `@mcp<sentry|events.search>`).
Current Kody Configuration
Review Options

The following review options are enabled or disabled:

Options Enabled
Bug ✅
Performance ✅
Security ✅
Business Logic ✅

Access your configuration settings here.

@codecov

codecov Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@27Bslash6
27Bslash6 merged commit b3a6c8a into main Oct 3, 2026
37 checks passed
@27Bslash6
27Bslash6 deleted the fix/LAB-7638-listener-warning-throttle branch October 3, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant