feat: Add usage budget controls by niemyjski · Pull Request #2270 · exceptionless/Exceptionless

niemyjski · 2026-05-31T04:55:16Z

Summary

Adds production-hardened organization budget alerts, explicit project ingest limits, automatic noisy-project sampling, and atomic ingest reservation accounting. The branch contains current origin/main, including the organization icon UI and controller coverage.

User-visible behavior

Organization Usage has a dedicated card for enabling, disabling, clearing, and previewing 1–99% budget alerts.
Project Usage has a dedicated card for fixed or percentage limits, effective-limit previews, clear support, and smart-throttle status.
Project charts prefer the effective project limit and identify whether the line is the project or organization limit.
Budget-alert and smart-throttle emails report current usage and the applicable threshold or fair-share limit.

Safety/compatibility

Existing organization overage, authorization, status-code, public API, and WebSocket behavior is preserved; new contract fields are additive.
JSON null clears only the two new settings objects; shared PATCH semantics are unchanged.
Unlimited/uncapped ingestion returns without cache or repository reads. Finite traffic uses bulk counter reads and one atomic reserve/complete pair.
Reservation records are generation-fenced and immutable to callers. Stale completion/release, malformed counters, identity mismatch, duplicate completion, replay, and compare-and-swap conflicts fail closed without partial accounting.
Usage totals, reserved capacity, blocked totals, reservation state, and usage-set membership are updated atomically in Redis. Scalar values retain per-key TTLs and list-key types are preflighted before mutation.
Smart throttling is transition-based, uses stable 5% sampling (including single-event posts), and counts each rejected event exactly once as blocked.
Alert/throttle claim and publication failures are isolated from ingestion. Delayed handlers recheck current settings, plan, usage, and throttle state.
No Elasticsearch mapping change or reindex is required.
Microsoft.OpenApi is pinned to 2.7.5 to clear GHSA-v5pm-xwqc-g5wc / CVE-2026-49451. dotnet list package --vulnerable --include-transitive reports no vulnerable NuGet packages.
Frontend package manifests and lockfile are unchanged from main; the existing static-build toolchain advisories reported by npm audit --omit=dev are not introduced by this PR.

Validation

dotnet build Exceptionless.slnx --no-restore --configuration Release: passed with 0 warnings and 0 errors.
Full backend suite: 2,594 total; 2,592 passed; 0 failed; 2 intentional performance-tool skips.
Focused usage-budget suite: 61/61 passed.
Event-post suite: 7/7 passed.
Real Redis atomic-batch suite: 5/5 passed, covering concurrent CAS, mismatches, per-key TTLs, sentinel-safe values, and wrong-type no-partial-write behavior.
npm ci, npm run check, npm run lint, npm run test:unit, and npm run build: passed.
openspec validate add-usage-budget-controls --strict --no-interactive: passed.
Changed-file formatting and git diff --check: passed.
GitHub CI: API/coverage, client, E2E, Docker build, version, and CLA are terminal green.

Local runtime dogfood

Aspire served exact head b696ef46de562819aee5bbd5b79dc80f102e2b47; API, Jobs, Redis, Elasticsearch, storage, and both web apps were healthy.
A fixed cap of 2 accepted 2/5 and blocked 3/5.
Two simultaneous 5-event requests against a cap of 6 accepted exactly 6 and blocked exactly 4, proving no oversubscription or lost blocked accounting through real Redis.
Clearing the first cap accepted 3 additional events while preserving the prior blocked count.
Aspire traces showed exactly two atomic batch scripts per concurrent ingest (reserve and complete), each completing in 0.35–0.77 ms. No error traces were associated with either test project.
The Svelte Usage form saved a limit change and immediately returned the matching effective limit. All related network requests were 200 and browser page errors were empty.
Desktop and 390×844 mobile layouts kept the budget controls readable and usable.
Both isolated dogfood projects were deleted afterward and verified absent with 404 responses.

Reviewer and architecture audit

Re-read all 37 review threads, all inline comments, 59 submitted reviews, and issue comments against the current effective diff.
32 resolved threads are outdated after subsequent rewrites; the remaining 5 resolved threads still attached to current lines are demonstrably fixed by current validation, nullable OpenAPI contracts, delayed-handler rechecks, and focused tests.
No unresolved review threads, change-request reviews, or actionable top-level reviewer comments remain.
The final thermo-nuclear review found no production file crossing 1,000 lines, no unowned feature branching, no thin ingest-store wrapper worth adding, no non-atomic usage transition, and no clearer code-judo simplification that preserves the current minimal boundary.

Rollout/kill switch

EnableSmartProjectThrottling defaults to true and can be set to false through configuration/environment for an immediate operational rollback. Explicit project limits and organization budget alerts remain available when the automatic sampler is disabled.

OpenSpec

openspec/changes/add-usage-budget-controls is complete: all 39 tasks are checked and strict validation passes.

Create proposal, design, tasks, and spec deltas for: - Organization budget alert emails - Automatic smart project throttling - Optional project event budgets - API compatibility - Organizations/projects/users/auth behavior Addresses customer feedback for budget alerts before overage and issue #112 requesting smarter project-level throttling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add OrganizationBudgetAlertSettings model with threshold validation - Add BudgetAlertSettings property to Organization model - Add ProjectIngestLimit model with Fixed and Percentage types - Add IngestLimit property to Project model - Create UpdateOrganization DTO for organization updates - Add BudgetAlertSettings to ViewOrganization - Add IngestLimit, EffectiveIngestLimit, IsSmartThrottled, SmartThrottleSampleRate to ViewProject - Update OrganizationController to use UpdateOrganization - Update Mapperly mappers for new properties Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add OrganizationBudgetAlert and ProjectSmartThrottleApplied messages - Add work items and handlers for budget alerts and smart throttle - Add CheckBudgetAlertThresholdsAsync to UsageService for threshold-based email alerts with per-billing-period deduplication - Add GetEventIngestAllowanceAsync for project-specific budget enforcement - Add GetSmartThrottleRateAsync for fair-share-based sampling - Integrate smart throttling and project budget limits into EventPostsJob - Add IMailer methods and Mailer implementations for budget alert and throttle notification emails - Add HTML email templates for both notification types - Register startup actions and work item handlers in bootstrappers - Update NullMailer and CountingMailer test doubles Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Wire up EffectiveIngestLimit, IsSmartThrottled, and SmartThrottleSampleRate in ProjectController.AfterResultMapAsync using UsageService methods. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Regenerate openapi.json to include UpdateOrganization, ProjectIngestLimit, OrganizationBudgetAlertSettings, and updated ViewProject/ViewOrganization schemas. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…hrottle Bug #1 (regression): OrganizationController.CanUpdateAsync unconditionally called IsOrganizationNameAvailableInternalAsync(changed.Name) without checking if Name was included in the PATCH delta. When only BudgetAlertSettings is sent, Name is null, causing the check to return false and reject the PATCH with a spurious duplicate-name error. Fix: Add ContainsChangedProperty(p => p.Name) guard, matching the pattern already used in ProjectController.CanUpdateAsync. Bug #2 (defensive): GetSmartThrottleRateAsync guarded maxEventsPerMonth < 0 but not == 0. If an organization is not found (cache miss + deleted org), GetMaxEventsPerMonthAsync returns 0, which would cause division by zero. In practice, line 508 (orgTotal <= 0) guard fires first for missing orgs, but the fix makes the invariant explicit and prevents any future path that reaches the division with maxEventsPerMonth=0. Fix: Change guard from < 0 to <= 0. Tests added: - PatchAsync_OnlyBudgetAlertSettings_DoesNotRejectDueToNameValidation (fails without Bug #1 fix, passes with it) - GetSmartThrottleRateAsync_OrganizationNotFound_ReturnsNoThrottle (defensive coverage for zero-limit path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds backend usage budget controls for organizations and projects, including budget alert settings, project ingest caps, smart throttling calculations, notification work items, and API contract updates.

Changes:

Adds organization budget alert and project ingest limit models/API fields.
Extends usage/event processing with alert checks, project budget allowance, and smart throttling sampling.
Adds notification messages, work items, mailer methods/templates, OpenSpec docs, OpenAPI baseline updates, and limited regression tests.

Reviewed changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
tests/Exceptionless.Tests/Services/UsageServiceTests.cs	Adds a regression test for missing organization smart throttle handling.
tests/Exceptionless.Tests/Mail/NullMailer.cs	Adds no-op implementations for new mailer methods.
tests/Exceptionless.Tests/Mail/CountingMailer.cs	Adds stub implementations for new mailer methods.
tests/Exceptionless.Tests/Controllers/OrganizationControllerTests.cs	Adds PATCH regression coverage for budget alert settings without name.
tests/Exceptionless.Tests/Controllers/Data/openapi.json	Updates OpenAPI baseline for new usage budget fields and DTOs.
src/Exceptionless.Web/Models/Project/ViewProject.cs	Exposes project ingest limit and smart throttle state.
src/Exceptionless.Web/Models/Project/UpdateProject.cs	Allows project ingest limit updates.
src/Exceptionless.Web/Models/Organization/ViewOrganization.cs	Exposes organization budget alert settings.
src/Exceptionless.Web/Models/Organization/UpdateOrganization.cs	Adds organization PATCH DTO.
src/Exceptionless.Web/Mapping/ProjectMapper.cs	Ignores computed project usage budget fields in mapper.
src/Exceptionless.Web/Mapping/OrganizationMapper.cs	Adds organization update mapping.
src/Exceptionless.Web/Controllers/ProjectController.cs	Populates computed project ingest/throttle fields.
src/Exceptionless.Web/Controllers/OrganizationController.cs	Switches update DTO and adjusts name validation condition.
src/Exceptionless.Web/Bootstrapper.cs	Registers new startup notification subscribers.
src/Exceptionless.Core/Services/UsageService.cs	Adds budget alert checks, project allowance calculation, and smart throttle calculation.
src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs	Adds smart throttle notification work item.
src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs	Adds budget alert notification work item.
src/Exceptionless.Core/Models/ProjectIngestLimit.cs	Adds project ingest limit model and enum.
src/Exceptionless.Core/Models/Project.cs	Persists optional project ingest limit.
src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs	Adds organization budget alert settings model.
src/Exceptionless.Core/Models/Organization.cs	Persists and validates organization budget alert settings.
src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs	Adds smart throttle notification message.
src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs	Adds budget alert notification message.
src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html	Adds smart throttle email template.
src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html	Adds budget alert email template.
src/Exceptionless.Core/Mail/Mailer.cs	Implements new budget/throttle emails.
src/Exceptionless.Core/Mail/IMailer.cs	Adds new budget/throttle mailer contracts.
src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs	Adds smart throttle subscriber and email handler.
src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs	Adds budget alert subscriber and email handler.
src/Exceptionless.Core/Jobs/EventPostsJob.cs	Enforces project allowance and applies smart throttle sampling.
src/Exceptionless.Core/Exceptionless.Core.csproj	Embeds new email templates.
src/Exceptionless.Core/Bootstrapper.cs	Registers new work item handlers.
openspec/changes/add-usage-budget-controls/tasks.md	Adds implementation/verification task plan.
openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md	Adds auth/ownership requirements.
openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md	Adds notification and queue requirements.
openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md	Adds ingestion, throttling, and project budget requirements.
openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md	Adds API compatibility requirements.
openspec/changes/add-usage-budget-controls/proposal.md	Documents proposal and user-visible behavior.
openspec/changes/add-usage-budget-controls/design.md	Documents detailed backend/UI design.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…t limits) - Org manage page: budget alert settings section with threshold enable/disable, add/remove threshold chips, and save via patchOrganization - Project manage page: event ingest limit section with type selector (none/fixed/percent), conditional numeric inputs, effective limit display, and smart throttle status indicator - New schemas: BudgetAlertSettingsSchema, UpdateOrganizationSchema, IngestLimitSchema, UpdateProjectIngestLimitSchema - New models: UpdateOrganization, OrganizationBudgetAlertSettingsUpdate, ProjectIngestLimitUpdate - Regenerated API types from baseline OpenAPI spec (includes new budget/limit types) - patchOrganization updated to use UpdateOrganization type for type safety Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bug fixes: - Wrap CheckBudgetAlertThresholdsAsync in try/catch so budget alert failures never break event ingestion - Use Math.Ceiling (not floor cast) for threshold event count and percentage project cap calculations - Add RecordSmartThrottleAsync to publish deduped ProjectSmartThrottleApplied notification (once per billing period per project) - Remove events.Count > 1 guard so single-event posts go through smart throttle - Fix IsOrganizationNameAvailableInternalAsync to accept excludeOrganizationId so PATCH with current name + budget_alert_settings doesn't fail as duplicate - Update OpenAPI baseline: budget_alert_settings and ingest_limit use proper $ref instead of untyped object schemas - Fix CanRunJob_WhenSmartThrottleApplied test: org needs MaxEventsPerMonth set to a finite value and a second project for projectCount > 1 condition Test coverage: - UsageServiceTests: 36 new tests covering GetEventIngestAllowanceAsync, CheckBudgetAlertThresholdsAsync, GetSmartThrottleRateAsync, percentage limit ceiling, and budget alert edge cases - ProjectControllerTests: ingest limit CRUD, validation, effective limit display, and smart throttle status - EventPostJobTests: project-over-limit blocking and smart throttle discard with regression assertions - OrganizationControllerTests: PATCH with same name + budget_alert_settings must not reject as duplicate name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… validation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add precondition assertion to verify smart throttle is active before running the job. Without warming the project-count cache first, the ES count query for GetOrganizationProjectCountAsync may return stale results causing the throttle to not fire. The pre-assertion also improves test documentation. Also improve assertion error messages with diagnostic detail.

BLOCKER 1 (frontend): addThreshold() accepted val=100 but backend rejects threshold >= 100 (valid range is 1-99). Fixed input max attribute and validation error message in org manage page. BLOCKER 2 (backend): non-atomic GetAsync+SetAsync for alertSentKey allowed concurrent workers to both see key absent and both publish duplicate alerts. Replaced with IncrementAsync (atomic); only the worker whose increment returns 1 (first writer wins) sends the OrganizationBudgetAlert message. Tests: adds regression test CheckBudgetAlertThresholdsAsync_ConcurrentCalls_PublishesAlertExactlyOnce which fires 5 concurrent IncrementTotalAsync calls and asserts exactly 1 alert is published. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 49 out of 52 changed files in this pull request and generated 14 comments.

…ction-hardening # Conflicts: # src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts # src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts # src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/manage/+page.svelte # src/Exceptionless.Web/Models/Project/UpdateProject.cs # tests/Exceptionless.Tests/Controllers/Data/openapi.json # tests/Exceptionless.Tests/Controllers/OrganizationControllerTests.cs

# Conflicts: # src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts # src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts # src/Exceptionless.Web/Controllers/OrganizationController.cs # src/Exceptionless.Web/Controllers/ProjectController.cs # tests/Exceptionless.Tests/Api/Data/openapi.json # tests/Exceptionless.Tests/Controllers/OpenApiControllerTests.cs

# Conflicts: # src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts # src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts

github-actions · 2026-07-26T15:33:38Z

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Core	74%	66%	10027	➖
Exceptionless.AppHost	38%	40%	139	❌
Exceptionless.Insulation	26%	27%	223	❌
Exceptionless.Web	84%	67%	6950	✔
Summary	77% (23453 / 30508)	66% (10806 / 16398)	17339	✔

niemyjski · 2026-07-26T16:16:28Z

Completed the final reviewer-feedback, thermo-nuclear, and runtime dogfood pass on b696ef46de562819aee5bbd5b79dc80f102e2b47.

Feedback audit: 37/37 review threads are resolved. 32 are outdated after later rewrites; the 5 still attached to current lines are already fixed by project-limit validation, nullable typed OpenAPI schemas, and delayed notification-handler rechecks. All 59 submitted reviews are COMMENTED; there are no change-request reviews, open threads to reply to, or actionable human/bot top-level comments.
Architecture: retained the minimal generic atomic-cache boundary rather than adding a thin ingest-store wrapper. Reservation generations, identity checks, accounting, blocked totals, capacity, state, TTLs, and usage-set membership are now handled atomically and fail closed. No changed production file crosses 1,000 lines and the final structural audit found no remaining code-judo, spaghetti, boundary, or partial-update blocker.
Local Redis dogfood: cap 2 produced 2 accepted / 3 blocked; two simultaneous 5-event requests against cap 6 produced exactly 6 accepted / 4 blocked; clearing the cap restored ingestion without altering prior blocked counts. Both isolated projects were deleted and verified absent.
Runtime cost/health: Aspire served the exact head; API/Jobs/dependencies were healthy. Each concurrent ingest used exactly two atomic scripts (reserve and complete), each 0.35–0.77 ms. No error traces were tied to the dogfood projects.
Browser: Svelte Usage saved 6 → 7 and returned effective_ingest_limit: 7; related requests were 200, page errors were empty, and desktop plus 390×844 layouts passed.
Verification: release build 0 warnings/errors; 2,592 passed / 2 skipped / 0 failed; focused usage 61/61, EventPost 7/7, real Redis 5/5; strict OpenSpec and diff checks passed. Required CI, including API coverage, is terminal green.

No external blocker remains.

niemyjski and others added 5 commits May 30, 2026 23:36

feat: populate ViewProject computed throttle properties

eb62767

Wire up EffectiveIngestLimit, IsSmartThrottled, and SmartThrottleSampleRate in ProjectController.AfterResultMapAsync using UsageService methods. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

chore: update OpenAPI baseline for new models

6f35269

Regenerate openapi.json to include UpdateOrganization, ProjectIngestLimit, OrganizationBudgetAlertSettings, and updated ViewProject/ViewOrganization schemas. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-code-quality Bot found potential problems May 31, 2026

View reviewed changes

Comment thread src/Exceptionless.Core/Models/Organization.cs Fixed

niemyjski marked this pull request as ready for review May 31, 2026 12:03

niemyjski requested a review from Copilot May 31, 2026 12:03

Copilot started reviewing on behalf of niemyjski May 31, 2026 12:04 View session

Copilot AI reviewed May 31, 2026

View reviewed changes

niemyjski and others added 2 commits May 31, 2026 07:16

github-code-quality Bot found potential problems May 31, 2026

View reviewed changes

Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated

Comment thread src/Exceptionless.Core/Services/UsageService.cs Fixed

niemyjski and others added 3 commits May 31, 2026 08:03

refactor: use LINQ Any() instead of foreach in Organization threshold…

5f97fa2

… validation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-code-quality Bot found potential problems May 31, 2026

View reviewed changes

Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated

niemyjski requested a review from Copilot May 31, 2026 23:19

Copilot started reviewing on behalf of niemyjski May 31, 2026 23:19 View session

Copilot AI reviewed May 31, 2026

View reviewed changes

niemyjski added 9 commits July 10, 2026 10:18

Harden usage budget controls for production

b15d293

Merge origin/main into niemyjski/add-usage-budget-controls

b81301c

fix: harden usage budget enforcement

e0bf535

fix: align usage budget client contracts

b9af005

Merge origin/main into niemyjski/add-usage-budget-controls

66e084d

# Conflicts: # src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts # src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts

refactor: centralize validation result mapping

cbd67a8

fix: harden usage reservation accounting

b696ef4

Uh oh!

Uh oh!

Conversation

niemyjski commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

User-visible behavior

Safety/compatibility

Validation

Local runtime dogfood

Reviewer and architecture audit

Rollout/kill switch

OpenSpec

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 26, 2026

Uh oh!

niemyjski commented Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

niemyjski commented May 31, 2026 •

edited

Loading