Repository navigation
docs(infrastructure): document the SSO gateway and how to protect an app with it #45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| { "title": "Guides", "pages": ["add-a-new-secret", "add-storage", "backups", "deploy-a-new-app"] } | ||
| { "title": "Guides", "pages": ["add-a-new-secret", "add-storage", "backups", "deploy-a-new-app", "protect-an-app-with-sso"] } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,168 @@ | ||
| --- | ||
| title: "Protect an App with SSO" | ||
| --- | ||
|
|
||
| ## Introduction | ||
|
|
||
| This guide puts an app behind the [SSO gateway](/docs/infrastructure/getting-started/cloud-infrastructure/sso): | ||
| visitors sign in with PoliNetwork Auth, and only people with a given permission get | ||
| through. Use it for apps without an OIDC login of their own. If the app supports OIDC, | ||
| give it its own client on Auth instead. | ||
|
|
||
| It assumes the app is already deployed and published on a domain | ||
| (see [Deploy a New App](/docs/infrastructure/Guides/deploy-a-new-app)). In the examples, | ||
| the namespace is `<k8s-namespace>` and the permission is `<app>:access`. | ||
|
|
||
| ## Procedure | ||
|
|
||
| 1. [Create the permission](#create-the-permission) on Auth | ||
| 2. [Add the Middleware](#add-the-middleware) | ||
| 3. [Use it from the Ingress](#use-it-from-the-ingress) | ||
| 4. [Admit only Traefik](#admit-only-traefik) | ||
| 5. Open a PR into polinetwork-cd, wait for CI, merge | ||
| 6. [Check it](#check-it), and only then disable the app's own login if you want to | ||
|
|
||
| ### Create the permission | ||
|
|
||
| On [auth.polinetwork.org](https://auth.polinetwork.org), under *Access → Permissions*, | ||
| create a permission named `<app>:access` (e.g. `uptime:access`). Then grant it to the roles | ||
| that should use the app, under *Access → Roles*. | ||
|
|
||
| You need `idp:permissions:write` and `idp:roles:write` for this; ask an IdP Admin | ||
| otherwise. No new application is needed: every app behind the gateway shares the | ||
| *SSO Gateway* client. | ||
|
|
||
| ### Add the Middleware | ||
|
|
||
| ```yaml title="apps/<k8s-namespace>/middleware.yaml" | ||
| apiVersion: traefik.io/v1alpha1 | ||
| kind: Middleware | ||
| metadata: | ||
| name: sso | ||
| spec: | ||
| forwardAuth: | ||
| address: http://oauth2-proxy.sso.svc.cluster.local:4180/?allowed_groups=<app>:access | ||
| trustForwardHeader: true | ||
| authResponseHeaders: | ||
| - X-Auth-Request-User | ||
| - X-Auth-Request-Email | ||
| - X-Auth-Request-Preferred-Username | ||
| addAuthCookiesToResponse: | ||
| - __Secure-pn_sso | ||
| - __Secure-pn_sso_0 | ||
| - __Secure-pn_sso_1 | ||
| - __Secure-pn_sso_2 | ||
| maxResponseBodySize: 1048576 | ||
| ``` | ||
|
|
||
| Change only the permission in `address`. The rest is the same for every app: | ||
|
|
||
| - `trustForwardHeader` passes the `https` scheme and the host to oauth2-proxy, so visitors | ||
| come back to the right page after signing in. | ||
| - `authResponseHeaders` tells the app who is signed in: `X-Auth-Request-User` and | ||
| `X-Auth-Request-Email` both carry the person's email address. Traefik also deletes | ||
| these headers from incoming requests, so visitors can't forge them. Only trust them in | ||
| an app that the `NetworkPolicy` below protects. | ||
| - `addAuthCookiesToResponse` sends the refreshed session cookie back to the browser. | ||
| **Don't remove it**: without it, sessions break after about 4 minutes. | ||
|
|
||
| <Callout type="note" title="One permission per Middleware"> | ||
|
|
||
| `allowed_groups` takes a comma-separated list, and **any one** permission is enough | ||
| (`?allowed_groups=a:access,b:access`). For different rules on different paths, add one | ||
| Middleware per rule, with one `Ingress` each. | ||
|
|
||
| </Callout> | ||
|
|
||
| Add the file to `apps/<k8s-namespace>/kustomization.yaml`, and make the app's Flux | ||
| Kustomization in `clusters/k3s/apps/<k8s-namespace>.yaml` wait for the gateway: | ||
|
|
||
| ```yaml title="clusters/k3s/apps/<k8s-namespace>.yaml" | ||
| dependsOn: | ||
| - name: infrastructure-secret-stores | ||
| - name: infrastructure-storage | ||
| - name: infrastructure-traefik | ||
| # [!code ++] | ||
| - name: infrastructure-sso | ||
| ``` | ||
|
|
||
| ### Use it from the Ingress | ||
|
|
||
| Reference the Middleware as `<k8s-namespace>-<name>@kubernetescrd`: | ||
|
|
||
| ```yaml title="apps/<k8s-namespace>/ingress.yaml" | ||
| metadata: | ||
| name: <app> | ||
| annotations: | ||
| traefik.ingress.kubernetes.io/router.entrypoints: web | ||
| # [!code ++] | ||
| traefik.ingress.kubernetes.io/router.middlewares: <k8s-namespace>-sso@kubernetescrd | ||
| ``` | ||
|
|
||
| The annotation applies to every host and path of that `Ingress`. Anything that must stay | ||
| public (a status page, a webhook, a health endpoint) goes in a **separate** `Ingress` | ||
| without the annotation, and that one should list exact paths, not `/`. See | ||
| `apps/uptime-kuma/ingress.yaml`: `uptime.polinetwork.org` needs `uptime:access`, while | ||
| `status.polinetwork.org` lists only the status page paths. | ||
|
|
||
| ### Admit only Traefik | ||
|
|
||
| The gateway protects the route, not the pod. Add a `NetworkPolicy` so nothing else in the | ||
| cluster can call the app's Service directly: | ||
|
|
||
| ```yaml title="apps/<k8s-namespace>/network-policy.yaml" | ||
| apiVersion: networking.k8s.io/v1 | ||
| kind: NetworkPolicy | ||
| metadata: | ||
| name: allow-traefik | ||
| spec: | ||
| podSelector: | ||
| matchLabels: | ||
| app: <app> | ||
| policyTypes: [Ingress] | ||
| ingress: | ||
| - from: | ||
| - namespaceSelector: | ||
| matchLabels: | ||
| kubernetes.io/metadata.name: kube-system | ||
| podSelector: | ||
| matchLabels: | ||
| app.kubernetes.io/name: traefik | ||
| ports: | ||
| - port: <port> | ||
| protocol: TCP | ||
| ``` | ||
|
|
||
| If other pods legitimately call the app (e.g. Prometheus scraping it), add them as further | ||
| `from` entries. | ||
|
|
||
| <Callout type="warning" title="Disabling the app's own login"> | ||
|
|
||
| Turn off the app's built-in login only once all of the following hold. Otherwise anyone | ||
| who reaches the app is an admin. | ||
|
|
||
| - The `NetworkPolicy` is applied. | ||
| - Every public `Ingress` lists only public paths. | ||
| - You have checked the gateway in a private window. | ||
|
|
||
| </Callout> | ||
|
|
||
| ### Check it | ||
|
|
||
| In a private window: | ||
|
|
||
| 1. Open the app. You should land on `auth.polinetwork.org`, and after signing in, back on | ||
| the page you opened. | ||
| 2. Sign in with an account **without** the permission: you should get `403 Forbidden`. | ||
| 3. Stay on the app for more than 5 minutes and keep using it: you shouldn't be sent back | ||
| to Auth. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| | Symptom | Likely cause | | ||
| |---------|--------------| | ||
| | `403` for everyone | The permission isn't granted to your role, or its name differs from `allowed_groups`. Check *Access → Roles* on Auth. | | ||
| | Back to the app on `http://` after sign-in | `trustForwardHeader: true` is missing from the Middleware. | | ||
| | Sent back to Auth every few minutes | `addAuthCookiesToResponse` is missing or doesn't list `__Secure-pn_sso`. | | ||
| | `404` on the app | The Middleware name or namespace in the annotation is wrong: Traefik drops the route. Check Traefik's logs in `kube-system`. | | ||
| | `500` on every protected app | oauth2-proxy is down: check `kubectl get pods -n sso` and the `infrastructure-sso` Kustomization in the Flux Web UI. | | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| { "title": "Cloud Infrastructure", "pages": ["terraform", "k3s-node", "flux"] } | ||
| { "title": "Cloud Infrastructure", "pages": ["terraform", "k3s-node", "flux", "sso"] } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| --- | ||
| title: "SSO Gateway" | ||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| Some apps have no login of their own, or one we'd rather not maintain (e.g. Uptime Kuma). | ||
| For those, Traefik asks an **SSO gateway** whether each request may go through. The | ||
| gateway signs people in with [PoliNetwork Auth](https://auth.polinetwork.org) and lets | ||
| through only people who hold the permission that app requires. | ||
|
|
||
| The gateway is [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/), deployed from | ||
| `infrastructure/sso` in [polinetwork-cd](https://github.com/PoliNetworkOrg/polinetwork-cd). | ||
| It replaces Cloudflare Access for HTTP apps. SSH to the node still goes through | ||
| Cloudflare Access for Infrastructure (see [Setup](/docs/infrastructure/getting-started/setup)). | ||
|
|
||
| <Callout type="info" title="Apps with their own OIDC login"> | ||
|
|
||
| Apps that can sign people in with OIDC themselves (Grafana, the Flux Web UI, Vaultwarden, | ||
| our own apps) keep doing so with their own client on PoliNetwork Auth. The gateway is for | ||
| everything else. | ||
|
|
||
| </Callout> | ||
|
|
||
| ## How a request goes through | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| participant B as Browser | ||
| participant T as Traefik | ||
| participant P as oauth2-proxy (sso) | ||
| participant A as auth.polinetwork.org | ||
| participant K as App | ||
| B->>T: GET https://uptime.polinetwork.org/ | ||
| T->>P: forwardAuth: /?allowed_groups=uptime:access | ||
| alt no session | ||
| P-->>T: 302 to auth.polinetwork.org | ||
| T-->>B: 302 | ||
| B->>A: sign in | ||
| A-->>B: 302 to sso.polinetwork.org/oauth2/callback | ||
| B->>P: callback (code) | ||
| P-->>B: session cookie on .polinetwork.org, 302 back to the app | ||
| else session without the permission | ||
| P-->>T: 403 | ||
| T-->>B: 403 | ||
| else session with the permission | ||
| P-->>T: 202 (+ refreshed cookie) | ||
| T->>K: original request | ||
| K-->>B: response | ||
| end | ||
| ``` | ||
|
|
||
| 1. The app's `Ingress` references a Traefik `forwardAuth` **Middleware**. Before routing | ||
| the request, Traefik sends its headers to oauth2-proxy at | ||
| `http://oauth2-proxy.sso.svc.cluster.local:4180/?allowed_groups=<permission>`. | ||
| 2. Without a session, oauth2-proxy redirects to PoliNetwork Auth (authorization code | ||
| flow with PKCE). After sign-in, Auth sends the browser back to | ||
| `https://sso.polinetwork.org/oauth2/callback`, which sets the session cookie and | ||
| returns the visitor to the page they asked for. | ||
| 3. With a session, oauth2-proxy checks the permission. It answers `202` and Traefik | ||
| forwards the request to the app, or `403` and the visitor sees an error page. | ||
|
|
||
| All protected apps share **one** client on Auth (*SSO Gateway*) and **one** redirect URI. | ||
| Adding an app doesn't touch Auth's application list: it only needs a permission. | ||
|
|
||
| ## Permissions | ||
|
|
||
| The Middleware names the permission in `allowed_groups`. oauth2-proxy reads it from the | ||
| `polinetwork_permission_list` claim, the array version of `polinetwork_permissions`, which | ||
| it requests with the `polinetwork:identity` scope. | ||
|
|
||
| - It checks **permissions**, never roles. The `groups` claim holds role keys and must not | ||
| be used for access. | ||
| - By convention, each app has its own `<app>:access` permission (e.g. `uptime:access`), | ||
| granted to the roles that should use it. Granting a role a permission is done on Auth, | ||
| not in Git. | ||
|
|
||
| ## Sessions | ||
|
|
||
| - **The cookie.** The session lives in an encrypted, `HttpOnly`, `Secure` cookie named | ||
| `__Secure-pn_sso`, set on `.polinetwork.org`. One sign-in covers every protected host. | ||
| Large sessions are split into `__Secure-pn_sso_0`, `_1`, and so on. Nothing is stored | ||
| server-side, so the two oauth2-proxy replicas are interchangeable. | ||
| - **Refresh.** Auth's ID and access tokens last 5 minutes. After 4 minutes, the next | ||
| request makes oauth2-proxy use the refresh token (`offline_access`) to get new ones, and | ||
| re-read the permissions. Losing a permission or a role takes effect within about 5 | ||
| minutes, without signing out. | ||
| - **Rotation.** Auth rotates the refresh token on every use, so the refreshed cookie | ||
| **must** reach the browser. That is what `addAuthCookiesToResponse` in the Middleware | ||
| does. Without it, the browser keeps sending a refresh token that has already been used. | ||
| - **Lifetime.** The cookie lasts 24 hours. After that, the visitor goes through Auth | ||
| again, usually without typing anything if their Auth session is still active. | ||
| - **Sign-out.** `https://sso.polinetwork.org/oauth2/sign_out` deletes the cookie. It does | ||
| not end the session on Auth. | ||
|
|
||
| ## Security notes | ||
|
|
||
| - **Only Traefik may reach a protected app.** The gateway protects the route, not the | ||
| pod. Every app behind it has a `NetworkPolicy` that admits only Traefik, or another pod | ||
| in the cluster could call its Service directly. This matters most for apps whose own | ||
| login is disabled. | ||
| - **Everything else on the app's host is public.** If a host serves both a private and a | ||
| public part, as Uptime Kuma does with its status page, the public part needs its own | ||
| `Ingress` that lists only the public paths. | ||
|
Comment on lines
+102
to
+104
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Describe public paths as explicit routes. “Everything else on the app’s host is public” is not the routing rule. Only paths matched by an Ingress without the SSO Middleware are public; unmatched paths are not exposed by that Ingress. The Uptime Kuma example also uses a separate
|
||
| - **It fails closed.** If oauth2-proxy is down, protected apps return errors; they don't | ||
| open up. | ||
| - **The cookie is a bearer credential.** Browsers send it to every `*.polinetwork.org` | ||
| host, including apps not behind the gateway. That is the price of one sign-in for all | ||
| apps; every host on the domain is ours. | ||
| - **Forwarded headers.** Traefik's `web` entrypoint trusts `X-Forwarded-*` from the pod | ||
| network, where `cloudflared` runs. oauth2-proxy uses `X-Forwarded-Proto` and | ||
| `X-Forwarded-Host` to send visitors back to `https://<host>/...`, and only to | ||
| `.polinetwork.org` hosts (`--whitelist-domain`). | ||
| - **Rate limits.** Refreshes call Auth's token endpoint from inside the cluster, so they | ||
| count against Auth's shared budget of 60 cluster-internal token requests per minute. | ||
| That's plenty for staff dashboards; don't put a high-traffic public app behind the | ||
| gateway. | ||
|
|
||
| ## Where things live | ||
|
|
||
| | What | Where | | ||
| |------|-------| | ||
| | oauth2-proxy Deployment, Service, Ingress, NetworkPolicy | `infrastructure/sso/` | | ||
| | Client secret and cookie secret | Key Vault `kv-pn-infra`: `sso-oidc-client-secret`, `sso-cookie-secret` | | ||
| | The *SSO Gateway* client | [auth.polinetwork.org/applications](https://auth.polinetwork.org/applications) | | ||
| | Trusted forwarded headers | `infrastructure/traefik/traefik-service.yaml` | | ||
| | Per-app Middleware (example) | `apps/uptime-kuma/middleware.yaml` | | ||
|
|
||
| To put an app behind the gateway, follow [Protect an App with SSO](/docs/infrastructure/Guides/protect-an-app-with-sso). | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: PoliNetworkOrg/docs
Length of output: 3359
Script executed:
Repository: PoliNetworkOrg/docs
Length of output: 15802
Web query:
Traefik ForwardAuth authResponseHeaders remove request headers before copying authentication response public route middleware documentationResult:
Script executed:
Repository: PoliNetworkOrg/docs
Length of output: 15802
Web query:
Traefik v3 ForwardAuth authResponseHeaders existing request headers router without middleware forwarded request headers defaultResult:
Clarify the header trust rule.
Public Ingresses bypass ForwardAuth. Requests through them can carry forged identity headers even when the NetworkPolicy is applied. This can affect apps that trust those headers on public paths.
Suggested fix
Committable suggestion