Skip to content

feat(occurrences): Add rollout utils#104841

Merged
thetruecpaul merged 1 commit intomasterfrom
cpaul/eap/add-rollout-utils
Dec 12, 2025
Merged

feat(occurrences): Add rollout utils#104841
thetruecpaul merged 1 commit intomasterfrom
cpaul/eap/add-rollout-utils

Conversation

@thetruecpaul
Copy link
Copy Markdown
Contributor

We want to control how reads from Occurrences on EAP happen. Generally, we want it to look like:

  • We roll out ingestion to some % of a region
  • (Optional pause to wait for some data)
  • We roll out double-read to the region
  • The snuba & EAP results will start different — because EAP doesn't have full retention for the full region yet — but should track to zero differences over time.

Then, for each new read we onboard:

  • Gate reading at all behind the standard rollout
  • Gate using the data behind the callsite allowlist
  • Set up a reasonable_match comparator
  • Land PR.
  • Observe data for callsite.
  • Should have 100% reasonable_match rate.
  • Should have exact_match rate go to 100% over time.
  • Once exact_match has been at 100% for some time, add callsite to allowlist.

@thetruecpaul thetruecpaul requested review from a team and shashjar December 12, 2025 05:14
@thetruecpaul thetruecpaul requested a review from a team as a code owner December 12, 2025 05:14
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Dec 12, 2025
We want to control how reads from Occurrences on EAP happen. Generally, we want it to look like:
* We roll out ingestion to some % of a region
* (Optional pause to wait for some data)
* We roll out double-read to the region
* The snuba & EAP results will start different — because EAP doesn't have full retention for the full region yet — but should track to zero differences over time.

Then, for each new read we onboard:
* Gate reading at all behind the standard rollout
* Gate using the data behind the callsite allowlist
* Set up a reasonable_match comparator
* Land PR.
* Observe data for callsite.
* Should have 100% reasonable_match rate.
* Should have exact_match rate go to 100% over time.
* Once exact_match has been at 100% for some time, add callsite to allowlist.
Comment on lines +8 to +9
def should_double_read_from_eap() -> bool:
return options.get("eap.occurrences.should_double_read")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
def should_double_read_from_eap() -> bool:
return options.get("eap.occurrences.should_double_read")

i think the option name is explanatory

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just prefer to centralize the rollout in case we want to do something smarter (e.g. have a callsite-level blocklist) in the future.

@thetruecpaul thetruecpaul merged commit 7610153 into master Dec 12, 2025
67 checks passed
@thetruecpaul thetruecpaul deleted the cpaul/eap/add-rollout-utils branch December 12, 2025 18:10
@github-actions github-actions bot locked and limited conversation to collaborators Dec 28, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants