feat: new session management

[lindesvard]

Summary by CodeRabbit

• New Features

  • Added automatic session cleanup via periodic reaper job
  • Added daily session vacuum to remove stale/orphaned sessions
  • Added comprehensive session lifecycle metrics and monitoring
  • Added end-to-end test suite for session handling and stress testing

• Bug Fixes

  • Improved session state resolution and idempotency handling
  • Fixed session boundary detection and lifecycle transitions

• Documentation

  • Added E2E test documentation with coverage scenarios and tunables

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 73effbf7-8e9a-4a73-a33b-724faae6d932

📥 Commits

Reviewing files that changed from the base of the PR and between 00d580e and 9240e7d.

📒 Files selected for processing (32)

• apps/api/e2e/README.md
• apps/api/e2e/lib.ts
• apps/api/e2e/session-e2e.ts
• apps/api/e2e/session-stress.ts
• apps/api/package.json
• apps/api/src/controllers/track.controller.ts
• apps/api/src/utils/ids.test.ts
• apps/api/src/utils/ids.ts
• apps/worker/src/boot-cron.ts
• apps/worker/src/boot-debug.ts
• apps/worker/src/jobs/cron.session-reaper.test.ts
• apps/worker/src/jobs/cron.session-reaper.ts
• apps/worker/src/jobs/cron.session-vacuum.test.ts
• apps/worker/src/jobs/cron.session-vacuum.ts
• apps/worker/src/jobs/cron.ts
• apps/worker/src/jobs/events.create-session-end.test.ts
• apps/worker/src/jobs/events.create-session-end.ts
• apps/worker/src/jobs/events.incoming-event.ts
• apps/worker/src/jobs/events.incoming-events.test.ts
• apps/worker/src/metrics.ts
• apps/worker/src/utils/session-handler.ts
• docker-compose.yml
• packages/db/scripts/check-sessions.ts
• packages/db/scripts/drain-old-session-jobs.ts
• packages/db/scripts/migrate-sessions.ts
• packages/db/src/buffers/base-buffer.ts
• packages/db/src/buffers/index.ts
• packages/db/src/buffers/session-buffer.test.ts
• packages/db/src/buffers/session-buffer.ts
• packages/db/src/services/event.service.ts
• packages/db/src/services/session.service.ts
• packages/queue/src/queues.ts

---

📝 Walkthrough

Walkthrough

This PR implements a comprehensive session lifecycle management system with boundary-aware session buffer, reaper/vacuum cron jobs for cleanup, idempotent session-end emission via Redis claims and v2 jobIds, device resolution using session liveness checks, and extensive E2E testing harness to validate end-to-end behavior.

Changes

Session Lifecycle Refactor

Layer / File(s)	Summary
Session Buffer Core: Ingest & Cleanup packages/db/src/buffers/session-buffer.ts, packages/db/src/buffers/index.ts, packages/db/src/buffers/base-buffer.ts	SessionBuffer.ingest() replaces add() to detect session boundaries using SESSION_TIMEOUT_MS, extends in-flight sessions with collapsing rows, or starts new sessions; getExistingSession() loads by (projectId, deviceId) or profile pointer; atomic CLEANUP_LUA script deletes blobs race-free; persistence writes Redis blob, wallclock ZSET, projects set, and ClickHouse buffer list.
Session Buffer Tests packages/db/src/buffers/session-buffer.test.ts	Tests refactored from add() to ingest() path; validates session creation, boundary detection, profile pointers, ClickHouse queuing, squash collapsing, and cleanup semantics.
Service Integration: Types & Event Persistence packages/db/src/services/event.service.ts, packages/db/src/services/session.service.ts, packages/queue/src/queues.ts	IClickhouseSession expanded with CollapsingMergeTree sign/version fields; createEvent() removes sessionBuffer.add() (delegates to callers); EventsQueuePayloadCreateSessionEnd includes snapshot; new CronQueuePayloadSessionReaper and CronQueuePayloadSessionVacuum types added.
Device/Session Resolution: EventTimeMs & Idle Windows apps/api/src/utils/ids.ts, apps/api/src/utils/ids.test.ts, apps/api/src/controllers/track.controller.ts	getDeviceId() accepts optional eventTimeMs and calls sessionBuffer.getExistingSession() to check liveness via ended_at against SESSION_TIMEOUT_MS; track controller threads event timestamp; tests validate deterministic session ID minting and salt-window behavior.
Incoming Event Lifecycle: Ingest → Boundary → Start/End apps/worker/src/jobs/events.incoming-event.ts, apps/worker/src/jobs/events.incoming-events.test.ts	Calls sessionBuffer.ingest() to determine canonical session state; enqueues session_end on boundary, emits session_start on new/boundary; propagates referrer fields from ingested session; tests use factory helpers to validate session_start emission counts.
Session Cleanup: Reaper (5m) & Vacuum (daily) apps/worker/src/jobs/cron.session-reaper.ts, apps/worker/src/jobs/cron.session-reaper.test.ts, apps/worker/src/jobs/cron.session-vacuum.ts, apps/worker/src/jobs/cron.session-vacuum.test.ts	sessionReaperCronJob scans wall-clock ZSET per project with advisory locks, enqueues session_end for idle sessions, handles orphans; sessionVacuumCronJob cleans stale/orphaned entries; both gated by environment feature flags and extensively tested.
Session-End Idempotency: V2 JobId & Redis Claims apps/worker/src/utils/session-handler.ts, apps/worker/src/jobs/events.create-session-end.ts, apps/worker/src/jobs/events.create-session-end.test.ts	getSessionEndJobIdV2() generates deterministic jobId keyed by session.id; enqueueSessionEndV2() includes session snapshot; createSessionEnd uses Redis SET NX claim for idempotency, skips if extended post-enqueue, selects live vs snapshot session for emission, performs cleanup; tests validate claim gating and session selection.
Cron Boot, Wiring & Session Metrics apps/worker/src/jobs/cron.ts, apps/worker/src/boot-cron.ts, apps/worker/src/boot-debug.ts, apps/worker/src/metrics.ts	Boots sessionReaper (5m) and sessionVacuum (daily 04:00 UTC); extends cron dispatcher and debug endpoints; exports session counters (starts/ends/reaps/vacuum) and histograms (duration/event counts); registers scrape-time gauges for active sessions, projects, and HWM lag via Redis queries.
E2E Testing Harness: Correctness & Stress apps/api/e2e/lib.ts, apps/api/e2e/session-e2e.ts, apps/api/e2e/session-stress.ts, apps/api/e2e/README.md, apps/api/package.json	lib.ts provides HTTP/Redis/ClickHouse helpers, polling, result tracking, and fixture setup; session-e2e.ts validates single-session lifecycle, boundary splits, replay, and identified profiles; session-stress.ts stress-tests concurrent sessions with drain-to-completion; scripts registered in npm and configurable via environment.
Migration & Health Checks packages/db/scripts/migrate-sessions.ts, packages/db/scripts/drain-old-session-jobs.ts, packages/db/scripts/check-sessions.ts	migrate-sessions.ts converts legacy session:{sessionId} to new session:{projectId}:{deviceId} via Redis pipeline; drain-old-session-jobs.ts removes legacy sessionEnd:* jobs while preserving v2; check-sessions.ts inspects Redis, queue, buffers, ClickHouse, and reports reconciliation issues with optional verbose sampling.
Infrastructure docker-compose.yml	Adds op-rp-console service (Redpanda Console v3.7.2) for local Kafka cluster observability on port 8091.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• Openpanel-dev/openpanel#302: Modifies session/device ID plumbing in track controller and introduces replay integration that this PR's E2E tests validate directly.
• Openpanel-dev/openpanel#231: Updates session end processing in createSessionEnd and session-handler, with this PR adding v2 idempotency on the same code paths.
• Openpanel-dev/openpanel#237: Introduces/modifies fetchDeviceId endpoint logic in track controller that directly overlaps with this PR's device resolution refactor.

Poem

A rabbit's delight in the session's new way,
Boundaries detected, old blobs swept away,
Reapers tick down, vacuums clean stray,
E2E tests affirm: "Everything's okay!" 🐇✨

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/new-session-management

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}