Skip to content

Add V3 streaming event ingestion pipeline#2370

Draft
ejsmith wants to merge 15 commits into
mainfrom
feature/v3-event-ingestion
Draft

Add V3 streaming event ingestion pipeline#2370
ejsmith wants to merge 15 commits into
mainfrom
feature/v3-event-ingestion

Conversation

@ejsmith

@ejsmith ejsmith commented Jul 13, 2026

Copy link
Copy Markdown
Member

Summary

  • add controller-free V3 Minimal API routes that stream top-level JSON values directly from HttpRequest.BodyReader using source-generated System.Text.Json metadata
  • move raw stack parsing and fingerprint generation to the server, with .NET, Java, JavaScript, Python, caused-by, manual-stacking, and normalized fallback support
  • resolve discarded stacks before quota reservation/materialization so discarded events take the cheapest path and are not charged
  • add distributed atomic quota reservation, deterministic event IDs, inline bulk persistence, ambiguous-write reconciliation, and a durable asynchronous side-effect work item
  • make stack statistics, notifications/webhooks, and post-save plugin work replay-safe with stable identities and per-event completion markers
  • add bounded microbatches, concurrency/backpressure, cancellation, gzip/Brotli support, compressed/decompressed/event/request limits, ProblemDetails, observability, and disabled-by-default allowlist rollout controls
  • publish an isolated V3 OpenAPI document, HTTP samples, architecture/rollout documentation, BenchmarkDotNet suites, and a repeatable streaming load harness
  • pin Microsoft.OpenApi 2.7.5 to remediate CVE-2026-49451 discovered during dependency validation

API and compatibility

V3 is a new breaking-contract surface:

  • POST /api/v3/events
  • POST /api/v3/projects/{projectId}/events
  • Content-Type: application/x-ndjson
  • one complete top-level JSON event per stream item; top-level arrays are rejected

The V2 controller, payload formats, queued handoff, and response contract remain available and unchanged. V3 is disabled by default and can be rolled out by project or organization allowlist.

Verification

  • dotnet build Exceptionless.slnx --no-restore --disable-build-servers -m:1 -v:minimal — 0 warnings, 0 errors
  • full backend suite — 2,350 passed, 0 failed, 2 intentional performance fixtures skipped
  • focused V3 persistence/replay/outbox suite — 26 passed
  • focused ingestion/compatibility suite — 85 passed
  • BenchmarkDotNet dry run — all 10 ingestion benchmarks executed
  • NuGet transitive vulnerability audit — no vulnerable packages in all eight projects
  • git diff --check — clean

Rollout

Keep this PR in draft while reviewing the architecture and running the documented one-, two-, four-, and eight-instance load matrix against the intended Redis/Elasticsearch topology. Production activation remains a configuration-only, reversible step.

Closes #2368

@ejsmith
ejsmith force-pushed the feature/v3-event-ingestion branch from 1a60290 to b3ce174 Compare July 13, 2026 22:48
@github-actions

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.Core 75% 67% 11130
Exceptionless.AppHost 38% 40% 139
Exceptionless.Insulation 45% 38% 308
Exceptionless.Web 84% 68% 7321
Summary 78% (25774 / 33211) 67% (11929 / 17902) 18898

@niemyjski

Copy link
Copy Markdown
Member

Follow-up feedback and thermo-nuclear audit complete.

Feedback inventory and classification:

  • 0 review threads, 0 inline review comments, and 0 submitted reviews from human, Codex, or Copilot reviewers. There were therefore no thread replies or resolutions to perform.
  • The sole conversation comment is the GitHub Actions coverage report (78% overall). It is informational and already passing, not an actionable review finding.

Thermo-nuclear full-diff result:

  • One structural blocker was found: UsageServiceTests.cs had grown from 658 to 1,183 lines because the V3 quota and settlement tests were appended to the existing broad fixture.
  • RCA: the new tests shared fixture state, so they accumulated in the original file instead of being separated by behavior.
  • Fix: commit 6d92911 splits the V3 quota and settlement cases into a focused partial test file without changing production code or test identities. The files are now 985 and 210 lines; no branch-added C# file crosses from below 1,000 lines to above it. The complete diff was also rechecked for spaghetti growth, boundary/type leaks, thin abstractions, duplicated helpers, sequential orchestration, and non-atomic updates; no other proven actionable defect remains.

Verification:

  • git diff --check: passed.
  • dotnet build --no-restore --disable-build-servers -m:1: passed with 0 warnings and 0 errors.
  • Test identity comparison: no method-name drift; all 29 UsageServiceTests cases are discovered after the split.
  • Local focused integration execution was blocked before test bodies ran by another worktree sharing the test-* Elasticsearch aliases (invalid_alias_name_exception). The isolated authoritative GitHub run is green: test-api 11m1s, test-e2e 5m7s, test-client 1m14s, docker-build 3m13s, website/version/CLA green.

No concrete external blocker remains. PR draft/admin state was intentionally left unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build a V3 streaming inline event ingestion pipeline

2 participants