Skip to content

Provide samples for using Python Connector with Django #437

Open

Description

@jackwotherspoon

Django is one of the most widely used Python web frameworks. We should look at providing samples on how to connect to a Cloud SQL database using the Python Connector and Django.

What makes this a more difficult feat is that the default Django backend for Postgres is psycopg2 and for MySQL is mysqlclient or mysql-connector-python, which are not currently supported with the Python Connector.

Activity

  1. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    type: docsImprovement to the documentation for an API.
    on Aug 25, 2022
  2. hadim commented on Nov 15, 2022

    @hadim

    Any ETA on this?

  3. jackwotherspoon commented on Nov 15, 2022

    @jackwotherspoon
    CollaboratorAuthor

    @hadim I can't really provide a precise ETA as of right now, it most likely won't be until early 2023. I have begun playing around with usage of the Python Connector and Django as is currently. However, if development work is needed to add support within Django for the Python Connector I won't be able to get around to it until the new year.

    If anyone here has had experience in leveraging the Python Connector with Django I'm more than happy to hear about their feedback/experience to update the Python Connector documentation accordingly with a code sample.

    Will ping this thread with any updates moving forward.

  4. added
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    and removed
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Dec 7, 2022
  5. stiangrim commented on Feb 17, 2023

    @stiangrim

    Any updates on this?

  6. jackwotherspoon commented on Feb 21, 2023

    @jackwotherspoon
    CollaboratorAuthor

    Hi @stiangrim, I've begun trying to find elegant workarounds to using the Cloud SQL Python Connector with Django as I hope to avoid having to write Django backend(s) to support the Python Connector.

    In the meantime if you or anyone else has had any luck feel free to post your finding here! I'm sure others would benefit greatly 😃

  7. danielroseman commented on Jul 28, 2023

    @danielroseman
    Contributor

    I don't know about doing it without writing a Django backend, but it turned out to be super easy to write one that supports the connector. In my case, since I was only targeting MySQL, I put this into a cloudsql/base.py file in my project:

    from django.db.backends.mysql import base
    from google.cloud.sql.connector import Connector
    
    
    class DatabaseWrapper(base.DatabaseWrapper):
        def get_new_connection(self, conn_params):
            return Connector().connect(**conn_params)
    

    and then with a DATABASES setting like this:

    DATABASES = {
      "default": {
        "ENGINE": "cloudsql",
        "USER": "...",
        "PASSWORD": "...",
        "NAME": "...",
        "OPTIONS": {
          "driver": "pymysql",
          "instance_connection_string": "project:region:instance"
        }
    }
    import pymysql
    pymysql.install_as_MySQLdb()
    

    it Just Worked. (The install_as_MySQLdb is needed as Django doesn't support PyMySQL out of the box; if the connector starts supporting mysqlclient, as proposed elsewhere, we can switch to that and this can go away.)

    Happy to contribute this as a PR to the docs.

  8. jackwotherspoon commented on Jul 28, 2023

    @jackwotherspoon
    CollaboratorAuthor

    @danielroseman That is great news! 👏

    Would be greatly appreciated if you could put up a PR adding this to the frameworks section of our README, we'd love to showcase this solution for others 😄

  9. enocom commented on Aug 10, 2023

    @enocom
    Member

    I don't know about doing it without writing a Django backend, but it turned out to be super easy to write one that supports the connector.

    Writing three separate custom backends for each db engine is probably the way to go. Incidentally, this mirrors what we do in Go with three separate database/sql hooks. Similar effort and similar result.

  10. 2 remaining items

  11. rcleveng commented on Mar 6, 2024

    @rcleveng

    Here's one way to do it.

    https://github.com/rcleveng/django_gcp_iam_auth

    Instructions on how to install with pip on the README.md
    Just update the DATABASES entry to replace the ENGINE for your connection.

    DATABASES["default"]["ENGINE"]  = "django_gcp_iam_auth.postgresql"
    
  12. enocom commented on Mar 6, 2024

    @enocom
    Member

    Thanks, @rcleveng -- that's a nice approach.

    In effect it's a variation of #214 (comment).

    It gets you IAM authentication without the Connector. Meanwhile, we're working on improving Django support in the near term.

  13. enocom commented on Mar 7, 2024

    @enocom
    Member

    Adding the code sample from the link above for posterity:

    import copy
    from django.db.backends.postgresql import base
    
    try:
        import google.auth
        import google.auth.exceptions
        from google.auth.transport import requests
        from google.auth.credentials import Credentials, Scoped, TokenState
    
    except google.auth.exceptions.DefaultCredentialsError:
        pass
    
    CLOUDSQL_IAM_LOGIN_SCOPE = ["https://www.googleapis.com/auth/sqlservice.login"]
    
    class DatabaseWrapper(base.DatabaseWrapper):
        def get_connection_params(self):
            params = super().get_connection_params()
            # need to remove this otherwise we'll get errors like
            #   'invalid dsn: invalid connection option "gcp_iam_auth"'
            if params.pop("gcp_iam_auth", None):
                self._credentials, _ = google.auth.default(scopes=CLOUDSQL_IAM_LOGIN_SCOPE)
                if not self._credentials.token_state == TokenState.FRESH:
                    self._credentials.refresh(requests.Request())
                params.setdefault("port", 5432)
                # TODO - should we add in a resource restriction for the DB instance?
                # https://cloud.google.com/iam/docs/downscoping-short-lived-credentials#auth_downscoping_token_broker-python
                # Set password to newly fetched access token
                params["password"] = self._credentials.token
    
            return params
  14. JoshTanke commented on Mar 24, 2024

    @JoshTanke

    Adding the code sample from the link above for posterity:

    import copy
    from django.db.backends.postgresql import base
    
    try:
        import google.auth
        import google.auth.exceptions
        from google.auth.transport import requests
        from google.auth.credentials import Credentials, Scoped, TokenState
    
    except google.auth.exceptions.DefaultCredentialsError:
        pass
    
    CLOUDSQL_IAM_LOGIN_SCOPE = ["https://www.googleapis.com/auth/sqlservice.login"]
    
    class DatabaseWrapper(base.DatabaseWrapper):
        def get_connection_params(self):
            params = super().get_connection_params()
            # need to remove this otherwise we'll get errors like
            #   'invalid dsn: invalid connection option "gcp_iam_auth"'
            if params.pop("gcp_iam_auth", None):
                self._credentials, _ = google.auth.default(scopes=CLOUDSQL_IAM_LOGIN_SCOPE)
                if not self._credentials.token_state == TokenState.FRESH:
                    self._credentials.refresh(requests.Request())
                params.setdefault("port", 5432)
                # TODO - should we add in a resource restriction for the DB instance?
                # https://cloud.google.com/iam/docs/downscoping-short-lived-credentials#auth_downscoping_token_broker-python
                # Set password to newly fetched access token
                params["password"] = self._credentials.token
    
            return params

    @enocom Your comments on this thread and others has been really helpful! I'm wondering if you might have any ideas on this related issue I'm hitting. I've been trying to get the IAM auth approach to work with Django on my local machine but the connection keeps timing out. AFAICT my psycopg2.connect database url matches the format in this other thead: dbname={db_name} client_encoding=UTF8 user={sa}@{project}.iam host={public_ip} port=5432 password={token}. I have public + private IPs enabled and also have "Require trusted client certificates" turned on in the SSL settings. My best guess is the SSL setting might be the issue - do I need to create an SSL client certificate and include it in my DatabaseWrapper somehow, or does the IAM auth flow do that for me somehow?

  15. jackwotherspoon commented on Mar 24, 2024

    @jackwotherspoon
    CollaboratorAuthor

    @JoshTanke Thanks for the comment on this issue. Do you mind creating a new issue on the repo with the error you are seeing? That way we can help debug your problem while keeping this thread clean 😄

    I have public + private IPs enabled and also have "Require trusted client certificates" turned on in the SSL settings. My best guess is the SSL setting might be the issue - do I need to create an SSL client certificate and include it in my DatabaseWrapper somehow, or does the IAM auth flow do that for me somehow?

    Once you open a new issue with your stacktrace I'll get a better grasp of what is going on. However, yes if you have "Require trusted client certificates" turned on and are not creating a database connection using SSL with client certs (mTLS). Which by the sample above and database URL you provided you are not then I would expect it to error. You would need to download your certificates and add their path to your psycopg2 database url (this SO post does a good job explaining the psycopg2 config)

    Depending on your use-case and especially for Private IP connections you may be fine with turning the setting to "Allow only SSL connections". This would allow you to update your database url with sslmode=require to:

    bname={db_name} client_encoding=UTF8 user={sa}@{project}.iam host={public_ip} port=5432 password={token} sslmode=require

    However, this still isn't ideal for Public IP connections which we are working on by adding support for psycopg2 to the Cloud SQL Python Connector soon (will be worked on this quarter) which will then also unblock Django for postgres and give users a way of using Django + Cloud SQL for "Require trusted client certificates" without having to manage certificates yourself.

  16. jjaikumar-xyla commented on Feb 22, 2025

    @jjaikumar-xyla

    Is support for psycopg2 still on the roadmap or is there a working code snippet? Would really love to use the Python Connector with my Django+Postgres via psycopg2 production application.

  17. jackwotherspoon commented on Feb 22, 2025

    @jackwotherspoon
    CollaboratorAuthor

    Is support for psycopg2 still on the roadmap or is there a working code snippet? Would really love to use the Python Connector with my Django+Postgres via psycopg2 production application.

    @jjaikumar-xyla psycopg2 support is in the backlog but not currently prioritized.

    I have been playing around a bit in my spare time with trying to get a PoC but still is a WIP.

    You can use Cloud SQL with psycopg2 + Django today via the workaround solution in the comments above in this thread:

    Here's one way to do it.

    https://github.com/rcleveng/django_gcp_iam_auth

    Instructions on how to install with pip on the README.md
    Just update the DATABASES entry to replace the ENGINE for your connection.

    DATABASES["default"]["ENGINE"] = "django_gcp_iam_auth.postgresql"

    Let me know if this works for you 😄. I will work on trying to get the team to prioritize psycopg2 support

  18. jjaikumar-xyla commented on Feb 22, 2025

    @jjaikumar-xyla

    Nice thanks for the update. My understanding if I'm reading the code correctly is that this approach doesn't use the proxy at all though right? So I would be in charge of implementing credential rotation for a long running service.

    Also since this isn't using the proxy, I still need to have my production network listed as an authorized network for the Cloud SQL instance correct?

  19. jackwotherspoon commented on Feb 23, 2025

    @jackwotherspoon
    CollaboratorAuthor

    Also since this isn't using the proxy, I still need to have my production network listed as an authorized network for the Cloud SQL instance correct?

    @jjaikumar-xyla Correct, it does not use the Proxy so if you are using public IP you would need to update authorized networks. I would recommend the solution above for private networks, if you are using a public IP Cloud SQL connection then Proxy is still your best alternative until we add psycopg2 support to the Python Connector.

  20. jjaikumar-xyla commented on Feb 23, 2025

    @jjaikumar-xyla

    Got it, super clear, thank you very much for the info! Would love to see psycopg2 support come through at some point.

  21. foxojs commented on Dec 12, 2025

    @foxojs

    Hi, came across this whilst attempting to migrate my django set up in cloud run from password based auth, to IAM, but struggling - I wondered if there is an approach that allows django to connect to postgres db yet via IAM user as was struggling to find anything? (I'm using postGIS)

  22. rcleveng commented on Dec 14, 2025

    @rcleveng

    Hi, came across this whilst attempting to migrate my django set up in cloud run from password based auth, to IAM, but struggling - I wondered if there is an approach that allows django to connect to postgres db yet via IAM user as was struggling to find anything? (I'm using postGIS)

    I wrote up
    https://github.com/rcleveng/django_gcp_iam_auth for that use case. I'm also close to releasing a new repo that can let you use iam auth with both sqlalchemy and psycopg3 pools (requires the new features in psycopg-pool v3.3 that came out December first, have been waiting for the callable support for conning)

    Note - that's not an official google project

  23. s-paquette commented on Feb 27, 2026

    @s-paquette

    We could really use an update on this for MySQL. Starting January next year MySQL 8.0 moves to paid extended support, but 8.4 still isn't supported for Django on App Engine Flex (or Cloud Run). The issue comes down to a lack of support for mysqlclient or mysql-connector-python by the CloudSQL Python Connector, which are what Django supports as engines. Django doesn't appear to have a roadmap or plan to support PyMySQL. There isn't at the moment any documented way to use Django's supported backends with the CloudSQL Python Connector, nor to use PyMySQL in conjunction with Dajngo and the CloudSQL Python Connector.

    An added wrinkle here is the CloudSQL Auth Proxy which is built in for App Engine Flex and Cloud Run doesn't seem to have been updated to support MySQL 8.4's default auth mechanism over UNIX sockets, and as far as I know Google's position is users running into this issue should move to the Connector. As indicated above, there's not a path for that at the moment for Django/MySQL users of those services.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.type: cleanupAn internal cleanup or hygiene concern.type: docsImprovement to the documentation for an API.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions