Skip to content

chore(agents): add capture message for missing required gen_ai attributes#104309

Merged
shellmayr merged 5 commits intomasterfrom
shellmayr/chore/capture-message-for-missing-genai-attributes
Dec 4, 2025
Merged

chore(agents): add capture message for missing required gen_ai attributes#104309
shellmayr merged 5 commits intomasterfrom
shellmayr/chore/capture-message-for-missing-genai-attributes

Conversation

@shellmayr
Copy link
Copy Markdown
Member

@shellmayr shellmayr commented Dec 3, 2025 •
edited
Loading

  • Add a capture message for the required attributes for client spans according to our documentation
  • Only capture when the origin is auto.ai.*
  • Add SDK version and name to the tags, so we know which versions are affected by this and we don't get alarmed by older versions exhibiting this behaviour
  • Also add the span id so we can investigate if something looks fishy

Closes TET-1529

@linear
Copy link
Copy Markdown

linear bot commented Dec 3, 2025

TET-1529 Log when span does not have required attributes

@github-actions github-actions bot added the Scope: Frontend Automatically applied to PRs that change frontend components label Dec 3, 2025
@codecov
Copy link
Copy Markdown

codecov bot commented Dec 3, 2025

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
12772 3 12769 10
View the top 3 failed test(s) by shortest run time 
getHighlightedSpanAttributes should not emit Sentry error when gen_ai span has all required attributes
Stack Traces | 0.005s run time 
Error: expect(jest.fn()).not.toHaveBeenCalled()

Expected number of calls: 0
Received number of calls: 1

1: "Gen AI span missing cost calculation", {"extra": {"attributes": {"gen_ai.agent.name": "my-agent", "gen_ai.operation.name": "chat", "gen_ai.origin": "auto.ai.openai", "gen_ai.request.model": "gpt-4", "gen_ai.system": "openai"}, "span_operation": "gen_ai.chat", "total_costs": undefined}, "level": "warning", "tags": {"feature": "agent-monitoring", "has_cost": "false", "has_model": "true", "model": "gpt-4", "span_operation": "gen_ai.chat", "span_type": "gen_ai"}}
    at Object.toHaveBeenCalled (.../traceDrawer/details/highlightedAttributes.spec.tsx:144:39)
    at Promise.finally.completed (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1559:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1499:10)
    at _callCircusTest (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1009:40)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at _runTest (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:949:3)
    at _runTestsForDescribeBlock (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:839:13)
    at _runTestsForDescribeBlock (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:829:11)
    at run (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:757:3)
    at runAndTransformResultsToJestFormat (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1920:21)
    at jestAdapter (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/runner.js:101:19)
    at runTestInternal (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:272:16)
    at runTest (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:340:7)
    at Object.worker (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:494:12)
getHighlightedSpanAttributes should use unknown for sdk_name and sdk_version when not provided
Stack Traces | 0.016s run time 
Error: expect(jest.fn()).toHaveBeenCalledWith(...expected)

- Expected
+ Received

  "Gen AI span missing required attributes",
@@ -2,11 +2,11 @@
    "level": "warning",
    "tags": Object {
      "feature": "agent-monitoring",
      "missing_attributes": "gen_ai.system,gen_ai.request.model,gen_ai.operation.name,gen_ai.agent.name",
      "origin": "auto.ai.openai",
-     "sdk_name": "unknown",
-     "sdk_version": "unknown",
+     "sdk": "unknown",
+     "span_id": "123",
      "span_operation": "gen_ai.chat",
      "span_type": "gen_ai",
    },
  },

Number of calls: 1
    at Object.toHaveBeenCalledWith (.../traceDrawer/details/highlightedAttributes.spec.tsx:188:35)
    at Promise.finally.completed (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1559:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1499:10)
    at _callCircusTest (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1009:40)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at _runTest (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:949:3)
    at _runTestsForDescribeBlock (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:839:13)
    at _runTestsForDescribeBlock (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:829:11)
    at run (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:757:3)
    at runAndTransformResultsToJestFormat (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1920:21)
    at jestAdapter (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/runner.js:101:19)
    at runTestInternal (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:272:16)
    at runTest (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:340:7)
    at Object.worker (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:494:12)
getHighlightedSpanAttributes should emit Sentry error when gen_ai span with auto.ai origin is missing required attributes
Stack Traces | 0.022s run time 
Error: expect(jest.fn()).toHaveBeenCalledWith(...expected)

Expected: "Gen AI span missing required attributes", {"level": "warning", "tags": {"feature": "agent-monitoring", "missing_attributes": "gen_ai.system,gen_ai.operation.name,gen_ai.agent.name", "origin": "auto.ai.openai", "sdk_name": "sentry.python", "sdk_version": "2.0.0", "span_operation": "gen_ai.chat", "span_type": "gen_ai"}}
Received
       1
          "Gen AI span missing cost calculation",
          Object {
        +   "extra": Object {
        +     "attributes": Object {
        +       "gen_ai.origin": "auto.ai.openai",
        +       "gen_ai.request.model": "gpt-4",
        +       "sdk.name": "sentry.python",
        +       "sdk.version": "2.0.0",
        +     },
        +     "span_operation": "gen_ai.chat",
        +     "total_costs": undefined,
        +   },
            "level": "warning",
            "tags": Object {
              "feature": "agent-monitoring",
        -     "missing_attributes": "gen_ai.system,gen_ai.operation.name,gen_ai.agent.name",
        -     "origin": "auto.ai.openai",
        -     "sdk_name": "sentry.python",
        -     "sdk_version": "2.0.0",
        +     "has_cost": "false",
        +     "has_model": "true",
        +     "model": "gpt-4",
              "span_operation": "gen_ai.chat",
              "span_type": "gen_ai",
            },
          },
       2
          "Gen AI span missing required attributes",
        @@ -2,11 +2,11 @@
            "level": "warning",
            "tags": Object {
              "feature": "agent-monitoring",
              "missing_attributes": "gen_ai.system,gen_ai.operation.name,gen_ai.agent.name",
              "origin": "auto.ai.openai",
        -     "sdk_name": "sentry.python",
        -     "sdk_version": "2.0.0",
        +     "sdk": "sentry.python@2.0.0",
        +     "span_id": "123",
              "span_operation": "gen_ai.chat",
              "span_type": "gen_ai",
            },
          },

Number of calls: 2
    at Object.toHaveBeenCalledWith (.../traceDrawer/details/highlightedAttributes.spec.tsx:112:35)
    at Promise.finally.completed (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1559:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1499:10)
    at _callCircusTest (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1009:40)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at _runTest (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:949:3)
    at _runTestsForDescribeBlock (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:839:13)
    at _runTestsForDescribeBlock (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:829:11)
    at run (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:757:3)
    at runAndTransformResultsToJestFormat (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/jestAdapterInit.js:1920:21)
    at jestAdapter (.../sentry/node_modules/.pnpm/jest-circus@30.0.4_babel-plugin-macros@3.1..../jest-circus/build/runner.js:101:19)
    at runTestInternal (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:272:16)
    at runTest (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:340:7)
    at Object.worker (.../sentry/node_modules/.pnpm/jest-runner@30.0..../jest-runner/build/testWorker.js:494:12)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@shellmayr shellmayr marked this pull request as ready for review December 3, 2025 10:03
@shellmayr shellmayr requested a review from a team as a code owner December 3, 2025 10:03
@shellmayr shellmayr requested a review from a team December 3, 2025 10:03
billyvg
billyvg reviewed Dec 3, 2025
View reviewed changes
static/app/views/performance/newTraceDetails/traceDrawer/details/highlightedAttributes.tsx
const sdkName = attributes['sdk.name'];
const sdkVersion = attributes['sdk.version'];

Sentry.captureMessage('Gen AI span missing required attributes', {
Copy link
Copy Markdown
Member

@billyvg billyvg Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use logs for this instead of captureMessage?

Copy link
Copy Markdown
Member Author

@shellmayr shellmayr Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We technically could, but for two reasons I'd prefer not to: a) we'd like counts of actual events for this, not extrapolated numbers and b) we have similar data that already uses captureMessage, and for sake of workflow and comparability, it makes sense to use the same approach.

@shellmayr shellmayr merged commit 152d002 into master Dec 4, 2025
49 checks passed
@shellmayr shellmayr deleted the shellmayr/chore/capture-message-for-missing-genai-attributes branch December 4, 2025 09:07
@github-actions github-actions bot locked and limited conversation to collaborators Dec 19, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@billyvg billyvg billyvg left review comments

@obostjancic obostjancic obostjancic approved these changes

Assignees

No one assigned

Labels

Scope: Frontend Automatically applied to PRs that change frontend components

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shellmayr @billyvg @obostjancic