Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/branch-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -379,6 +379,10 @@ jobs:
agent_sandbox_version: v0.5.0
topology: sidecar
extra_helm_values: deploy/helm/openshell/ci/values-sidecar.yaml
- agent_sandbox_api: v1beta1
agent_sandbox_version: v0.5.0
topology: proxy-pod
extra_helm_values: deploy/helm/openshell/ci/values-proxy-pod.yaml
permissions:
actions: read
contents: read
Expand Down
3 changes: 3 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions crates/openshell-driver-kubernetes/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
openshell-otel = { path = "../openshell-otel" }
openshell-policy = { path = "../openshell-policy" }

Expand All @@ -37,6 +38,8 @@ tracing-subscriber = { workspace = true }
thiserror = { workspace = true }
miette = { workspace = true }
notify = "8"
rand = { workspace = true }
rcgen = { workspace = true }

[dev-dependencies]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
Expand Down
53 changes: 53 additions & 0 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,59 @@ this driver. Kubernetes owns scheduling and pod lifecycle. The
`openshell-sandbox` supervisor inside each workload owns agent isolation,
credential injection, policy polling, logs, and the gateway relay.

### RFC 0012 proxy-pod topology

Set `topology = "proxy-pod"` and explicitly set
`proxy_pod.network_policy_enforced = true` after verifying that the cluster CNI
enforces `networking.k8s.io/v1` `NetworkPolicy`. The workload pod runs
`openshell-sandbox --mode=boundary`; a separately scheduled Deployment runs
`openshell-sandbox --mode=control` and owns gateway, provider, policy, and
upstream credentials.

The driver creates the workload egress fence before the Sandbox CR and creates
the CR in a suspended state. It then observes the namespace, Sandbox,
Deployment, and fence identities; creates one immutable bootstrap Secret with
separately mounted boundary and control records; and starts control before
releasing the workload. The boundary/control TCP protocol uses a per-sandbox CA
and server certificate in addition to its bootstrap-token authentication.

The workload `NetworkPolicy` denies every workload-initiated egress connection
and permits ingress only from the uniquely paired control pod to the configured
boundary port. The workload receives no gateway endpoint, gateway TLS secret,
projected service-account token, provider identity socket, or upstream-proxy
credential. Its boundary resolves process and socket-owner identity locally and
streams attributed connections to control. Control performs destination and L7
policy evaluation and makes the actual upstream connection.

Stop suspends the workload before scaling control to zero. Start restores
control before resuming the workload. Delete retains the unowned egress fence
until workload-pod deletion is confirmed; Kubernetes garbage collection removes
the Sandbox-owned Secret, Service, Deployment, and control policy.
The control Deployment becomes Available only after its TCP readiness probe
observes that control has attached, confirmed enforcement, started or resumed
the boundary process, and installed the access plane. The driver combines that
availability with the boundary Service and workload-fence presence when
publishing sandbox readiness. Periodic reconciliation repairs control replica
drift. It suspends the workload if an immutable-identity companion is deleted
and cannot be reconstructed without changing the boundary's trusted claims.
The control container runs as the namespace-resolved non-root UID/GID with no Linux
capabilities, a read-only root filesystem, and the runtime-default seccomp
profile. Orphan cleanup retains newly created workload fences for at least five
minutes so it cannot race the fence-first Sandbox creation sequence.
If gateway failure interrupts initial companion creation, reconciliation keeps
the Sandbox suspended and rolls back the unchanged partial CR after five
minutes. Bootstrap Secrets remain create-only to avoid granting the gateway
read access to arbitrary Secrets in operator namespaces. A deleted Secret does
not affect an already-mounted control; any replacement pod fails its semantic
readiness probe and the driver reports the Sandbox not ready.

This posture requires more than accepting a `NetworkPolicy` object. Policies
are additive, and Kubernetes does not expose portable CNI enforcement
attestation. The sandbox namespace must therefore be controlled so untrusted
principals cannot create permissive policies, create pods, read bootstrap
Secrets, or spoof the pair labels. The explicit acknowledgement is an operator
contract, not runtime CNI detection.

## Sandbox Resource

The driver works with the `agents.x-k8s.io` `Sandbox` custom resource. It
Expand Down
67 changes: 65 additions & 2 deletions crates/openshell-driver-kubernetes/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,13 +72,17 @@ pub enum SupervisorTopology {
/// Run network supervision in a privileged sidecar and process supervision
/// as a low-capability wrapper in the agent container.
Sidecar,
/// Run the shared RFC 0012 control role in a separate pod and the boundary
/// role beside the workload.
ProxyPod,
}

impl std::fmt::Display for SupervisorTopology {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Combined => f.write_str("combined"),
Self::Sidecar => f.write_str("sidecar"),
Self::ProxyPod => f.write_str("proxy-pod"),
}
}
}
Expand All @@ -90,11 +94,47 @@ impl FromStr for SupervisorTopology {
match s {
"combined" => Ok(Self::Combined),
"sidecar" => Ok(Self::Sidecar),
"proxy-pod" => Ok(Self::ProxyPod),
other => Err(format!("unknown topology '{other}'")),
}
}
}

/// Driver-owned requirements for the cross-pod RFC 0012 topology.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct KubernetesProxyPodConfig {
/// Explicit operator assertion that the cluster CNI enforces
/// `networking.k8s.io/v1` `NetworkPolicy` for the sandbox namespaces.
pub network_policy_enforced: bool,
/// TCP port exposed by the workload boundary to its paired control pod.
pub boundary_port: u16,
}

impl Default for KubernetesProxyPodConfig {
fn default() -> Self {
Self {
network_policy_enforced: false,
boundary_port: 5500,
}
}
}

impl KubernetesProxyPodConfig {
pub fn validate(&self) -> Result<(), String> {
if !self.network_policy_enforced {
return Err(
"proxy-pod topology requires proxy_pod.network_policy_enforced = true after the operator has verified CNI NetworkPolicy enforcement"
.to_string(),
);
}
if self.boundary_port == 0 {
return Err("proxy_pod.boundary_port must be nonzero".to_string());
}
Ok(())
}
}

/// How workspaces map to Kubernetes namespaces.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
Expand Down Expand Up @@ -326,6 +366,8 @@ pub struct KubernetesComputeConfig {
pub topology: SupervisorTopology,
/// Sidecar-only settings used when `topology = "sidecar"`.
pub sidecar: KubernetesSidecarConfig,
/// Cross-pod boundary settings used when `topology = "proxy-pod"`.
pub proxy_pod: KubernetesProxyPodConfig,
/// Corporate HTTP forward proxy used by the network supervisor for
/// policy-approved TLS CONNECT egress.
pub https_proxy: Option<String>,
Expand Down Expand Up @@ -451,6 +493,7 @@ impl Default for KubernetesComputeConfig {
supervisor_sideload_method: SupervisorSideloadMethod::default(),
topology: SupervisorTopology::default(),
sidecar: KubernetesSidecarConfig::default(),
proxy_pod: KubernetesProxyPodConfig::default(),
https_proxy: None,
no_proxy: None,
proxy_auth_secret_name: None,
Expand Down Expand Up @@ -503,7 +546,11 @@ impl KubernetesComputeConfig {
}

pub fn validate_proxy_uid(&self) -> Result<(), String> {
self.sidecar.validate_proxy_uid()
self.sidecar.validate_proxy_uid()?;
if self.topology == SupervisorTopology::ProxyPod {
self.proxy_pod.validate()?;
}
Ok(())
}

/// Validate the operator-owned corporate upstream proxy configuration.
Expand Down Expand Up @@ -580,7 +627,7 @@ impl KubernetesComputeConfig {
}
if self.topology == SupervisorTopology::Combined {
return Err(
"proxy credential Secrets require topology = \"sidecar\"; combined topology shares the credential mount with the workload and fsGroup can make it readable by the sandbox user"
"proxy credential Secrets require topology = \"sidecar\" or \"proxy-pod\"; combined topology shares the credential mount with the workload and fsGroup can make it readable by the sandbox user"
.to_string(),
);
}
Expand Down Expand Up @@ -946,6 +993,22 @@ mod tests {
assert_eq!(cfg.topology, SupervisorTopology::Combined);
}

#[test]
fn proxy_pod_requires_network_policy_enforcement_acknowledgement() {
let mut cfg = KubernetesComputeConfig {
topology: SupervisorTopology::ProxyPod,
..KubernetesComputeConfig::default()
};
assert!(
cfg.validate_proxy_uid()
.unwrap_err()
.contains("network_policy_enforced")
);

cfg.proxy_pod.network_policy_enforced = true;
cfg.validate_proxy_uid().unwrap();
}

#[test]
fn serde_rejects_sidecar_binary_identity_field() {
let json = serde_json::json!({
Expand Down
Loading
Loading