Certificate issues

Note
See this comment for the latest troubleshooting information and fixes.

VS Code uses a version of Electron that does not support registering self-signed intermediate certificates with extensions on Windows. This unfortunately blocks our ability to perform HTTPS requests securely to a Coder deployment with self-signed certificates.

See electron/electron#38527 for updates an information. We'll fix this as soon as we can!

Warning
"unable to verify the first certificate" is the error message you'll see.

@kylecarbs

It seems there are still cases where allow-insecure: true does not allow the extension to connect. @kylecarbs is investigating this.

How are folks reproducing this? I tried a self-signed certificate and an untrusted authority but both work on Windows after I add the relevant certificates to the system (I am doing so via mmc). I tested against the following:

https://untrusted-root.badssl.com
https://self-signed.badssl.com
https://localhost:8081 with a Caddy instance proxying to dev.coder.com using a self-signed certificate

All three cases work after I add the certificates (and restart VS Code) so VS Code is indeed reading certificates installed on the system (here by "works" I mean the request goes through without any certificate errors---since the first two are not actual Coder deployments logging in fails with an "invalid token" error).

But those three cases give me variants of "self signed certificate" errors, not the reported error. This domain can trigger the "unable to verify the first certificate" error though:

https://incomplete-chain.badssl.com

However this ("incomplete chain") implies to me that the server is misconfigured and nothing can be done on the client to fix this?

I could not get the coder binary nor Firefox to work against this domain either but the report seem to suggest that people are getting this error while the coder binary still works. If I can figure out how to reproduce that specifically I can continue debugging.

Going to take a look at the Electron bug in the meantime.

When I generate my own self-signed certificate it works for the Electron test case. This leads me to believe that there is a way to generate a certificate that is accepted by Node but not Electron, maybe because Node is missing a check that should cause it to reject the certificate.

Either way, so far this seems to imply that the issue is a certificate that is invalid in some way. It could be helpful to understand how people are generating the certificates that result in this error so I can reproduce a case where it does work for the coder binary, curl, etc, but not for VS Code.

Another note: I am able to reproduce "unable to get local issuer certificate" in VS Code 1.78.1. This is a bug that was fixed in 1.78.2 but it does not match the reported error so it might be irrelevant anyway. However, when getting a reproduction we should be careful to note the VS Code version just in case.

Figured out a way to reproduce:

Create a certificate on Windows via Powershell using the New-SelfSignedCertificate command
Export the certificate via mmc
Convert from .pfx to .pem files using openssl

The resulting certificate and key will trigger the Electron error, and if I use them in Caddy then Firefox, Chrome, and the coder binary all accept the certificate but in VS Code I get the "unable to verify the first certificate" error, as expected since Electron gives the same error.

So we come back to needing to figure out why Electron's behavior is different and what is different about these certificates compared to generating them with openssl, for example. The fix might either be in Electron or in certificate generation, or both.

Edit: here is a way to generate a certificate that reproduces the issue using openssl.

    openssl req -x509 -nodes -newkey rsa:2048 -sha256 -days 365 \
            -keyout localhost.key -out localhost.crt \
            -addext "keyUsage = digitalSignature, keyEncipherment" \
            -addext "extendedKeyUsage = clientAuth, serverAuth" \
            -addext "authorityKeyIdentifier = none" \
            -addext "subjectAltName=DNS:localhost" \
            -subj "/CN=localhost"

If you add -addext "basicConstraints = CA:FALSE" then it will work on Firefox as well (but still not in Electron). The default is CA:TRUE. The Windows-generated certificate has no basic constraints listed at all but I could not figure out how to omit that using openssl.

More details can be found in the Electron issue.

@code-asher

Un-assigning @code-asher in favor of #115 since that is specifically tracking the investigation of the root cause of the bug since it's a little uncertain where in the upstream dependency chain things are breaking.

@code-asher

@code-asher gave me a great run-down of what is going on here (and helped with edits to this summary). There are 2 issues that users are running into that we are aware of.

Case 1) Partial certificate chain on server

You are likely running into this if Coder warns you the certificate is self-signed, and when you press "allow insecure," you get stuck in an infinite refresh loop and Coder doesn't let you connect.

Here's why:

golang, curl, and Google Chrome all accept only a "leaf" certificate if it is trusted on the client machine (so they do not require any intermediates or roots)
node requires a leaf + intermediate certificate server certs which must be signed by a root certificate trusted by the system

The fix for Coder admins: Update the certificates on the server to include both the leaf certificate + intermediate cert. You can cat them together into one file, starting at the leaf and going up the chain (so leaf first, then next intermediate, and so on)

To improve the UX: The Coder extension should attempt to detect (by parsing the server certificate) whether it is a partial chain. If so, we should hide the "allow insecure" button (since it does not help in this case). Instead, we link to docs on a GitHub issue on how to change the server certificate to work with node. #116

If it's not easy to detect if the problem is case 1 or case 2, we should link to both issues in the "insecure" popup that the extension creates.

Case 2) A certificate does not have signing capabilities

You are likely running into this if Coder warns you the certificate is self-signed, and when you press "allow insecure," you are able to connect to the workspace 🎉

Here's why:

openssl accept these certificates anyways
boringssl, the SSL library that node uses does not accept certificates if the system certificate does not have signing capabilities.

More details + the fix for Coder users and admins are outlined here: #115 (comment)

To improve the UX: We are working on a patch upstream to boringssl + Electron (electron/electron#38527), but this will take some time. In the meantime, the "allow insecure" button should work in the pop-up dialog or you can follow the docs above to regenerate your local certificates with signing capabilities.

@kylecarbs

Hey @kylecarbs @bpmct just checking to see if any permanent solution has been found for this issue? We are evaluating coder for our platform, eventually enterprise, and ran into this issue when launching vscode from coder. I saw @code-asher's reply #115 but unfortunately we work in a restricted environment and our CA will not allow the coder server's certificate to have keyUsage: Certificate Sign, so we are without a way around it. We have loaded our entire bundle onto the coder server and tried allow insecure but no luck.

Has the root certificate also been added to the client machine's trust store? Unfortunately the allow insecure option is bugged, for some reason it just does nothing in VS Code so that is something we need to investigate, but for now it means the certificates need to be trusted on the client.

I opened #151

@code-asher

hey @code-asher thanks for the follow-up. Yes the root and intermediates are installed on the client's trust store. Tried adding those to the coder image as a test as well, but am still met with the error saying "Your certificate is not capable of signing...".

Ah OK bummer. That does seem weird though. That error happens when we get unable to verify leaf signature, and that error happens when the leaf certificate is the only one available (if I am reading the code right).

Does it look like the certificates are properly chained? Like the leaf certificate has an issuer and key id fields pointing to the next certificate, etc. For example (some parts omitted for brevity):

coder@dev27:~/vscode-coder/fixtures$ openssl x509 -text -noout -in chain-leaf.crt 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = TEST-intermediate
        Validity
            Not Before: Oct 13 20:35:09 2023 GMT
            Not After : Sep 19 20:35:09 2123 GMT
        Subject: CN = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
        X509v3 extensions:
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:localhost
            X509v3 Subject Key Identifier: 
                5A:7A:13:6C:1C:53:64:9E:A9:D8:C5:D3:BF:8A:3C:29:94:59:1E:90
            X509v3 Authority Key Identifier: 
                12:32:21:AD:6E:A1:48:3E:36:77:DB:DF:78:DB:53:D4:4C:FA:79:A7

If we can figure out how to generate a chain similar to the ones used in your setup I can attempt to reproduce.

Closing this out as we'd have to push a fix to boringssl and this has limited scope. We can re-open if we hear more feedback on this in the future.

@bpmct

@bpmct @code-asher @kylecarbs

Could you maybe explain in more detail (which config setting) is being referenced here.

The fix for Coder admins: Update the certificates on the server to include both the leaf certificate + intermediate cert. You can cat them together into one file, starting at the leaf and going up the chain (so leaf first, then next intermediate, and so on)

Specifically which certificates? The CA bundle? The servers TLS certs? If this in reference to the CA cert bundle? This setting? https://github.com/coder/coder/blob/53e8e9c7cd5a24d42e5c51e6b6335ad8be692db9/helm/coder/values.yaml#L210 If so i have my full CA bundle set there (wondering if there might be a different case I am hitting), and still seem to be getting this error which I believe puts me in case 1):

PARTIAL_CHAIN = "Your Coder deployment's certificate cannot be verified because a certificate is missing from its chain. To fix this your deployment's administrator should bundle the missing certificates or you can add the missing certificates directly to this system's trust store."

#105 (comment)

Specifically which certificates? The CA bundle? The servers TLS certs? If this in reference to the CA cert bundle?

The "fix for Coder admins" line is in reference to the server's TLS certificates. So, if my deployment was coder.com, it would be in reference to the certificate(s) you would get if you were to run openssl s_client -showcerts -connect coder.com:443 for example.

Once the site is serving the full chain you then need to make sure the root certificate is trusted in both the workspace and your local machine, if you are using an internal CA or something.

@code-asher

@code-asher

Thanks! This got me fixed

I had already included my full chained CA bundle for Coder with this config setting https://github.com/coder/coder/blob/53e8e9c7cd5a24d42e5c51e6b6335ad8be692db9/helm/coder/values.yaml#L210 which is what I thought it was asking for but obviously wasn't.

Our k8s cluster is set up where our ingress controller has a wild card cert, which only had the "leaf". I am calling it a "leaf" as that is the dns name the cert is created for. So we had something like the following in the crt file and that was it.

leaf - *.k8s.example.com

The fix (which I didn't realize was something you could even do...) was to make the crt file.

leaf - *.k8s.example.com
intermediate CA crt
root CA crt

I put this file on our ingress controller and Coder started working.

One thing I am confused about is the difference between

https://github.com/coder/coder/blob/53e8e9c7cd5a24d42e5c51e6b6335ad8be692db9/helm/coder/values.yaml#L319

and

https://github.com/coder/coder/blob/53e8e9c7cd5a24d42e5c51e6b6335ad8be692db9/helm/coder/values.yaml#L156

So in my case I’m using my k8s clusters ingress controller itself to provide the TLS termination via wild cards URLs. If I wasn’t doing that my solution would be to provide my full root chain cert to line 319 (which I didn't test but I assume would accomplish the same thing), but i’m unclear on what line 156 is doing.

Perfect yup, the root certificate is optional, but no harm in adding it to the bundle either (the relevant RFC https://www.rfc-editor.org/rfc/rfc5246#section-7.4.2 for anyone curious).

I could be wrong, but I think 319 would be doing TLS termination with NGINX or whatever and 156 would be doing TLS termination with Coder itself rather than a separate ingress controller.

Navigation Menu

Repository navigation

Certificate issues #105

Case 1) Partial certificate chain on server

Case 2) A certificate does not have signing capabilities

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Certificate issues #105

Description

Activity

bpmct commented on Jun 14, 2023

code-asher commented on Jul 6, 2023

code-asher commented on Jul 6, 2023

code-asher commented on Jul 6, 2023

code-asher commented on Jul 7, 2023

sreya commented on Jul 11, 2023

bpmct commented on Jul 21, 2023

Case 1) Partial certificate chain on server

Case 2) A certificate does not have signing capabilities

pjmckee commented on Oct 8, 2023

code-asher commented on Oct 11, 2023

pjmckee commented on Oct 12, 2023

code-asher commented on Oct 13, 2023

Kira-Pilot commented on May 15, 2024

vaskokj commented on May 22, 2025

code-asher commented on May 22, 2025

vaskokj commented on May 23, 2025

code-asher commented on May 27, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions