Skip to content

fix: retry transient network errors during OAuth token refresh#987

Draft
devin-ai-integration[bot] wants to merge 2 commits intomainfrom
devin/1776051256-fix-oauth-connection-error-retry
Draft

fix: retry transient network errors during OAuth token refresh#987
devin-ai-integration[bot] wants to merge 2 commits intomainfrom
devin/1776051256-fix-oauth-connection-error-retry

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

Summary

The CDK's OAuth token refresh mechanism (_make_handled_request in abstract_oauth.py) did not retry on transient network errors like ConnectionError, ConnectTimeout, or ReadTimeout. The @backoff.on_exception decorator only retried on DefaultBackoffException (raised for HTTP 429/5xx responses). Transient network errors — such as ConnectionResetError during SSL handshake — fell through the exception handler and were re-raised without retry, producing a generic message like:

An unexpected error occurred in stream deals_property_history: ConnectionError

This was classified as system_error instead of transient_error, which could prevent automatic retry at the platform level and cause unnecessary oncall alerts.

Changes

  1. Added transient network exceptions to the backoff retry decoratorrequests.exceptions.ConnectionError, ConnectTimeout, and ReadTimeout are now retried with exponential backoff (up to 300s), consistent with TRANSIENT_EXCEPTIONS in rate_limiting.py which already handles these for regular HTTP stream requests.

  2. Wrapped exhausted-retry network errors in AirbyteTracedException — When retries are exhausted, refresh_access_token catches the transient exception and wraps it in AirbyteTracedException with FailureType.transient_error and a clear user-facing message: "OAuth access token refresh request failed due to a network error."

  3. Wrapped generic exceptions with AirbyteTracedException — The catch-all except Exception block in _make_handled_request now raises AirbyteTracedException with FailureType.system_error instead of a bare Exception, ensuring consistent structured error reporting.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/16179

Review & Testing Checklist for Human

  • Verify the backoff decorator tuple includes the right exception types and doesn't accidentally retry non-transient errors
  • Confirm the refresh_access_token wrapper correctly catches only transient network exceptions (not all RequestException subclasses)
  • Check that the generic except ExceptionAirbyteTracedException(system_error) change in _make_handled_request doesn't alter behavior for existing exception types that should propagate differently
  • Run a connector that uses OAuth (e.g., source-hubspot) and verify token refresh still works normally
  • Simulate a ConnectionError during token refresh to confirm retry + proper error message

Notes

Link to Devin session: https://app.devin.ai/sessions/c8765b364f2d49c6a3d6e94361bba420

Add ConnectionError, ConnectTimeout, and ReadTimeout to the backoff
retry decorator in _make_handled_request so transient network failures
during OAuth token refresh are automatically retried.

When retries are exhausted, wrap the exception in AirbyteTracedException
with FailureType.transient_error and a clear user-facing message instead
of exposing raw Python exception class names.

Also wrap generic exceptions in _make_handled_request with
AirbyteTracedException and FailureType.system_error for consistent
structured error reporting.

Co-Authored-By: bot_apk <apk@cognition.ai>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1776051256-fix-oauth-connection-error-retry#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1776051256-fix-oauth-connection-error-retry

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

PyTest Results (Fast)

4 017 tests  +5   4 006 ✅ +5   7m 25s ⏱️ -18s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit db51c89. ± Comparison against base commit 4aaafcf.

♻️ This comment has been updated with latest results.

The test_cloud_read_with_private_endpoint test asserted the old generic
Exception message. Updated to match the new AirbyteTracedException
message from _make_handled_request.

Co-Authored-By: bot_apk <apk@cognition.ai>
@github-actions
Copy link
Copy Markdown

PyTest Results (Full)

4 020 tests  +5   4 008 ✅ +5   11m 12s ⏱️ +13s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit db51c89. ± Comparison against base commit 4aaafcf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants