Skip to content

bug: Kubernetes sandbox crashes EROFS — recursive chown /sandbox fails on read-only submounts (0.0.82) #2294

Closed

Description

@bjw123

Summary

On the Kubernetes driver, the sandbox supervisor performs a recursive chown /sandbox ("Chowning /sandbox for driver-injected UID/GID") during startup. If a read-only volume mount exists at any path under /sandbox (a common pattern for making a workspace subdirectory immutable), the recursive chown hits that read-only mount and aborts with EROFS: Read-only file system. The agent process never starts and the pod goes CrashLoopBackOff.

The same pod spec (with read-only submounts under the workspace) worked on our previous version (~0.0.39-era); it began failing after upgrading to 0.0.82.

Environment

  • OpenShell 0.0.82 — gateway ghcr.io/nvidia/openshell/gateway:0.0.82 + its bundled/version-pinned supervisor
  • Kubernetes compute driver (in_tree=true)
  • Sandbox pods run under gVisor (runsc RuntimeClass)
  • server.disableTls: true; policy has landlock.compatibility: best_effort

Steps to reproduce

  1. Create a sandbox whose pod mounts a readOnly: true volume (e.g. an emptyDir) at a subpath of the workspace — for example /sandbox/.openclaw/skills — to make that directory immutable.
  2. Start the sandbox on 0.0.82 (Kubernetes driver).
  3. The agent container crash-loops.

Actual behavior

Supervisor log (protocol noise trimmed); chown /sandbox is the last real operation before teardown:

INFO  openshell_supervisor_process::process: Updated /etc/passwd and /etc/group for sandbox identity
INFO  openshell_supervisor_process::process: Chowning /sandbox for driver-injected UID/GID
DEBUG openshell_supervisor_process::netns: Cleaning up network namespace       # teardown starts
OCSF  CONFIG:CLEANED_UP [INFO] Network namespace cleaned up
Error:   × EROFS: Read-only file system
stream closed: EOF (agent)

The error is a bare EROFS with no path; it correlates exactly with a readOnly mount located under /sandbox. Moving that same read-only mount outside /sandbox (and symlinking it in — chown -R doesn't descend into symlinks / out-of-tree targets) makes the sandbox start normally, which confirms the recursive chown as the cause.

Expected behavior

Chowning a read-only mount is expected to fail; it should not be fatal. The recursive workspace chown should do one of:

  • skip paths on read-only filesystems (detect the read-only mount / EROFS and continue), or
  • not descend into nested mount points (don't cross filesystem boundaries during the recursive chown), or
  • treat a per-entry EROFS as a non-fatal warning.

Why this matters (gVisor)

Making specific workspace subdirectories immutable via read-only mounts is the only option under gVisor: the filesystem_policy read_only mechanism relies on Landlock, which runsc does not implement (OCSF FINDING [HIGH] "Landlock Filesystem Sandbox Unavailable"). So read-only mounts are the sole gVisor-compatible way to enforce per-directory immutability — and they are exactly what this recursive chown breaks.

Workaround

Relocate the read-only mounts outside /sandbox and symlink the in-workspace paths to them (create the symlinks before the agent starts). The recursive chown /sandbox then leaves them alone, and the agent still sees a read-only directory. Verified working on 0.0.82, but it shouldn't be necessary.

Related

Activity

  1. elezar commented on Jul 15, 2026

    @elezar
    Member

    📋 triage-agent

    Triage Assessment

    Classification: bug-confirmed

    Summary

    This matches a real Kubernetes sandbox startup failure. The Kubernetes driver injects OPENSHELL_SANDBOX_UID / OPENSHELL_SANDBOX_GID, and the supervisor then recursively chowns /sandbox; a read-only submount under /sandbox can therefore return EROFS before the workload starts.

    Investigation

    Recommendation

    Route this as a confirmed sandbox/Kubernetes bug. A fix should make the /sandbox ownership pass avoid crossing nested mounts and/or handle per-entry EROFS as a warning instead of aborting the supervisor. Please add focused supervisor coverage for the ownership walk behavior where possible, plus Kubernetes e2e coverage for a readOnly submount under /sandbox.

    Do not apply state:agent-ready until a maintainer explicitly approves implementation.

  2. added
    area:sandboxSandbox runtime and isolation work
    area:clusterRelated to running OpenShell on k3s/docker
    test:e2e-kubernetesRequires Kubernetes end-to-end coverage
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Jul 15, 2026
  3. added theissue type on Jul 15, 2026
  4. varshaprasad96 commented on Jul 16, 2026

    @varshaprasad96
    Contributor

    @elezar I can get a PR up to fix this issue!

  5. bjw123 commented on Jul 16, 2026

    @bjw123
    ContributorAuthor

    i am also happy to if my vouch request could be approved, i made one a while back #1829

    maybe initial one was too much as roadmap was not defined but it would be nice to be able to contribute to upstream

  6. varshaprasad96 commented on Jul 17, 2026

    @varshaprasad96
    Contributor

    @bjw123 thanks for raising this bug, would you like to collaborate on this issue? I have created a PR #2341, would be great to get some eyes. Thanks!

  7. added a commit that references this issue on Jul 17, 2026
    90fae13
  8. bjw123 commented on Jul 20, 2026

    @bjw123
    ContributorAuthor

    Thank you @varshaprasad96 i did some validation testing and this MR does fix the bug

  9. added a commit that references this issue on Jul 22, 2026
    8a14b3a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:clusterRelated to running OpenShell on k3s/dockerarea:sandboxSandbox runtime and isolation worktest:e2e-kubernetesRequires Kubernetes end-to-end coverage

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions