Skip to content

Add orphan cleanup for cloud resources - #4401

Merged
r4victor merged 42 commits into
masterfrom
issue_3453_orphaned_instances
Oct 22, 2026
Merged

r4victor merged 42 commits into
masterfrom
issue_3453_orphaned_instances

Conversation

@r4victor

@r4victor r4victor commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #3453

dstack could leave cloud resources running if

  • the server crashed between creating them and recording them
  • the provider failed after creating them
  • termination gave up
  • there is a dstack or provider bug.

Solution:

  • Instances get a provider_ref that backends set on the resources they create, and every issued provider_ref is recorded in a new provider_refs table. Records outlive deleted instances, such as job placeholder instances deleted after a failed run_job(), so their leftover resources are still recognized.
  • Backends opt in by implementing the ComputeWithResourceListingSupport mixin, which lists resources marked with provider_ref and deletes them.
  • AWS implements it as an example: instances and Elastic IPs are tagged and listed. Other backends will be added in separate PRs.
  • A scheduled task enrolls backends of projects with issued provider_refs, and OrphanCleanupPipeline periodically lists those of active projects, matches resources to instances by provider_ref, and reports or deletes orphans, emitting events.
  • DSTACK_SERVER_ORPHAN_CLEANUP_MODE is report by default (log, Sentry and events only), with delete and off as alternatives.
  • See contributing/ORPHAN-CLEANUP.md for the design.

Limitations:

  • Only AWS supports orphan cleanup for now.
  • Compute groups, placement groups, volumes and gateways are not covered yet.
  • On dstack Sky, each project's dstack backend lists the shared base backend accounts, so listings repeat per project. The plan is to clean up each shared account once and match resources across projects.
  • Rows in provider_refs are never deleted yet. The plan is to delete rows of deleted or terminated instances after a retention period.
  • delete should become the default only after running report in production and adding a circuit breaker that stops deletion when unexpectedly many orphans are found.

@r4victor
r4victor marked this pull request as ready for review October 22, 2026 11:05
@r4victor
r4victor merged commit 6361caf into master Oct 22, 2026
29 checks passed
@r4victor
r4victor deleted the issue_3453_orphaned_instances branch October 22, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Detect orphan instances

1 participant