You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Provide samples for using Python Connector with Django #437
Django is one of the most widely used Python web frameworks. We should look at providing samples on how to connect to a Cloud SQL database using the Python Connector and Django.
Reacted by Hadrien Mary, Phillip Cutter, Aly Sivji, pjohn, pasvel, Loïc Motheu, Miles Thibault, Stian Tornholm Grimsgaard, Kaan, Akseli Nelander and 9 more
@hadim I can't really provide a precise ETA as of right now, it most likely won't be until early 2023. I have begun playing around with usage of the Python Connector and Django as is currently. However, if development work is needed to add support within Django for the Python Connector I won't be able to get around to it until the new year.
If anyone here has had experience in leveraging the Python Connector with Django I'm more than happy to hear about their feedback/experience to update the Python Connector documentation accordingly with a code sample.
Will ping this thread with any updates moving forward.
Hi @stiangrim, I've begun trying to find elegant workarounds to using the Cloud SQL Python Connector with Django as I hope to avoid having to write Django backend(s) to support the Python Connector.
In the meantime if you or anyone else has had any luck feel free to post your finding here! I'm sure others would benefit greatly 😃
Reacted by Stian Tornholm Grimsgaard and Jean Carrière
I don't know about doing it without writing a Django backend, but it turned out to be super easy to write one that supports the connector. In my case, since I was only targeting MySQL, I put this into a cloudsql/base.py file in my project:
from django.db.backends.mysql import base
from google.cloud.sql.connector import Connector
class DatabaseWrapper(base.DatabaseWrapper):
def get_new_connection(self, conn_params):
return Connector().connect(**conn_params)
it Just Worked. (The install_as_MySQLdb is needed as Django doesn't support PyMySQL out of the box; if the connector starts supporting mysqlclient, as proposed elsewhere, we can switch to that and this can go away.)
Happy to contribute this as a PR to the docs.
Reacted by Stian Tornholm Grimsgaard, Jim Angel, Aly Sivji and Ezio Tamburini
Would be greatly appreciated if you could put up a PR adding this to the frameworks section of our README, we'd love to showcase this solution for others 😄
I don't know about doing it without writing a Django backend, but it turned out to be super easy to write one that supports the connector.
Writing three separate custom backends for each db engine is probably the way to go. Incidentally, this mirrors what we do in Go with three separate database/sql hooks. Similar effort and similar result.
Adding the code sample from the link above for posterity:
importcopyfromdjango.db.backends.postgresqlimportbasetry:
importgoogle.authimportgoogle.auth.exceptionsfromgoogle.auth.transportimportrequestsfromgoogle.auth.credentialsimportCredentials, Scoped, TokenStateexceptgoogle.auth.exceptions.DefaultCredentialsError:
passCLOUDSQL_IAM_LOGIN_SCOPE= ["https://www.googleapis.com/auth/sqlservice.login"]
classDatabaseWrapper(base.DatabaseWrapper):
defget_connection_params(self):
params=super().get_connection_params()
# need to remove this otherwise we'll get errors like# 'invalid dsn: invalid connection option "gcp_iam_auth"'ifparams.pop("gcp_iam_auth", None):
self._credentials, _=google.auth.default(scopes=CLOUDSQL_IAM_LOGIN_SCOPE)
ifnotself._credentials.token_state==TokenState.FRESH:
self._credentials.refresh(requests.Request())
params.setdefault("port", 5432)
# TODO - should we add in a resource restriction for the DB instance?# https://cloud.google.com/iam/docs/downscoping-short-lived-credentials#auth_downscoping_token_broker-python# Set password to newly fetched access tokenparams["password"] =self._credentials.tokenreturnparams
Adding the code sample from the link above for posterity:
importcopyfromdjango.db.backends.postgresqlimportbasetry:
importgoogle.authimportgoogle.auth.exceptionsfromgoogle.auth.transportimportrequestsfromgoogle.auth.credentialsimportCredentials, Scoped, TokenStateexceptgoogle.auth.exceptions.DefaultCredentialsError:
passCLOUDSQL_IAM_LOGIN_SCOPE= ["https://www.googleapis.com/auth/sqlservice.login"]
classDatabaseWrapper(base.DatabaseWrapper):
defget_connection_params(self):
params=super().get_connection_params()
# need to remove this otherwise we'll get errors like# 'invalid dsn: invalid connection option "gcp_iam_auth"'ifparams.pop("gcp_iam_auth", None):
self._credentials, _=google.auth.default(scopes=CLOUDSQL_IAM_LOGIN_SCOPE)
ifnotself._credentials.token_state==TokenState.FRESH:
self._credentials.refresh(requests.Request())
params.setdefault("port", 5432)
# TODO - should we add in a resource restriction for the DB instance?# https://cloud.google.com/iam/docs/downscoping-short-lived-credentials#auth_downscoping_token_broker-python# Set password to newly fetched access tokenparams["password"] =self._credentials.tokenreturnparams
@enocom Your comments on this thread and others has been really helpful! I'm wondering if you might have any ideas on this related issue I'm hitting. I've been trying to get the IAM auth approach to work with Django on my local machine but the connection keeps timing out. AFAICT my psycopg2.connect database url matches the format in this other thead: dbname={db_name} client_encoding=UTF8 user={sa}@{project}.iam host={public_ip} port=5432 password={token}. I have public + private IPs enabled and also have "Require trusted client certificates" turned on in the SSL settings. My best guess is the SSL setting might be the issue - do I need to create an SSL client certificate and include it in my DatabaseWrapper somehow, or does the IAM auth flow do that for me somehow?
@JoshTanke Thanks for the comment on this issue. Do you mind creating a new issue on the repo with the error you are seeing? That way we can help debug your problem while keeping this thread clean 😄
I have public + private IPs enabled and also have "Require trusted client certificates" turned on in the SSL settings. My best guess is the SSL setting might be the issue - do I need to create an SSL client certificate and include it in my DatabaseWrapper somehow, or does the IAM auth flow do that for me somehow?
Once you open a new issue with your stacktrace I'll get a better grasp of what is going on. However, yes if you have "Require trusted client certificates" turned on and are not creating a database connection using SSL with client certs (mTLS). Which by the sample above and database URL you provided you are not then I would expect it to error. You would need to download your certificates and add their path to your psycopg2 database url (this SO post does a good job explaining the psycopg2 config)
Depending on your use-case and especially for Private IP connections you may be fine with turning the setting to "Allow only SSL connections". This would allow you to update your database url with sslmode=require to:
However, this still isn't ideal for Public IP connections which we are working on by adding support for psycopg2 to the Cloud SQL Python Connector soon (will be worked on this quarter) which will then also unblock Django for postgres and give users a way of using Django + Cloud SQL for "Require trusted client certificates" without having to manage certificates yourself.
Is support for psycopg2 still on the roadmap or is there a working code snippet? Would really love to use the Python Connector with my Django+Postgres via psycopg2 production application.
Is support for psycopg2 still on the roadmap or is there a working code snippet? Would really love to use the Python Connector with my Django+Postgres via psycopg2 production application.
@jjaikumar-xyla psycopg2 support is in the backlog but not currently prioritized.
I have been playing around a bit in my spare time with trying to get a PoC but still is a WIP.
You can use Cloud SQL with psycopg2 + Django today via the workaround solution in the comments above in this thread:
Nice thanks for the update. My understanding if I'm reading the code correctly is that this approach doesn't use the proxy at all though right? So I would be in charge of implementing credential rotation for a long running service.
Also since this isn't using the proxy, I still need to have my production network listed as an authorized network for the Cloud SQL instance correct?
Also since this isn't using the proxy, I still need to have my production network listed as an authorized network for the Cloud SQL instance correct?
@jjaikumar-xyla Correct, it does not use the Proxy so if you are using public IP you would need to update authorized networks. I would recommend the solution above for private networks, if you are using a public IP Cloud SQL connection then Proxy is still your best alternative until we add psycopg2 support to the Python Connector.
Hi, came across this whilst attempting to migrate my django set up in cloud run from password based auth, to IAM, but struggling - I wondered if there is an approach that allows django to connect to postgres db yet via IAM user as was struggling to find anything? (I'm using postGIS)
Hi, came across this whilst attempting to migrate my django set up in cloud run from password based auth, to IAM, but struggling - I wondered if there is an approach that allows django to connect to postgres db yet via IAM user as was struggling to find anything? (I'm using postGIS)
I wrote up https://github.com/rcleveng/django_gcp_iam_auth for that use case. I'm also close to releasing a new repo that can let you use iam auth with both sqlalchemy and psycopg3 pools (requires the new features in psycopg-pool v3.3 that came out December first, have been waiting for the callable support for conning)
We could really use an update on this for MySQL. Starting January next year MySQL 8.0 moves to paid extended support, but 8.4 still isn't supported for Django on App Engine Flex (or Cloud Run). The issue comes down to a lack of support for mysqlclient or mysql-connector-python by the CloudSQL Python Connector, which are what Django supports as engines. Django doesn't appear to have a roadmap or plan to support PyMySQL. There isn't at the moment any documented way to use Django's supported backends with the CloudSQL Python Connector, nor to use PyMySQL in conjunction with Dajngo and the CloudSQL Python Connector.
An added wrinkle here is the CloudSQL Auth Proxy which is built in for App Engine Flex and Cloud Run doesn't seem to have been updated to support MySQL 8.4's default auth mechanism over UNIX sockets, and as far as I know Google's position is users running into this issue should move to the Connector. As indicated above, there's not a path for that at the moment for Django/MySQL users of those services.
Activity
hadim commentedon Nov 15, 2022
Any ETA on this?
jackwotherspoon commentedon Nov 15, 2022
@hadim I can't really provide a precise ETA as of right now, it most likely won't be until early 2023. I have begun playing around with usage of the Python Connector and Django as is currently. However, if development work is needed to add support within Django for the Python Connector I won't be able to get around to it until the new year.
If anyone here has had experience in leveraging the Python Connector with Django I'm more than happy to hear about their feedback/experience to update the Python Connector documentation accordingly with a code sample.
Will ping this thread with any updates moving forward.
stiangrim commentedon Feb 17, 2023
Any updates on this?
jackwotherspoon commentedon Feb 21, 2023
Hi @stiangrim, I've begun trying to find elegant workarounds to using the Cloud SQL Python Connector with Django as I hope to avoid having to write Django backend(s) to support the Python Connector.
In the meantime if you or anyone else has had any luck feel free to post your finding here! I'm sure others would benefit greatly 😃
danielroseman commentedon Jul 28, 2023
I don't know about doing it without writing a Django backend, but it turned out to be super easy to write one that supports the connector. In my case, since I was only targeting MySQL, I put this into a
cloudsql/base.pyfile in my project:and then with a DATABASES setting like this:
it Just Worked. (The
install_as_MySQLdbis needed as Django doesn't support PyMySQL out of the box; if the connector starts supporting mysqlclient, as proposed elsewhere, we can switch to that and this can go away.)Happy to contribute this as a PR to the docs.
jackwotherspoon commentedon Jul 28, 2023
@danielroseman That is great news! 👏
Would be greatly appreciated if you could put up a PR adding this to the frameworks section of our README, we'd love to showcase this solution for others 😄
Add docs for running with Django
Add docs for connecting to MySQL with Django.
enocom commentedon Aug 10, 2023
Writing three separate custom backends for each db engine is probably the way to go. Incidentally, this mirrors what we do in Go with three separate
database/sqlhooks. Similar effort and similar result.2 remaining items
rcleveng commentedon Mar 6, 2024
Here's one way to do it.
https://github.com/rcleveng/django_gcp_iam_auth
Instructions on how to install with pip on the README.md
Just update the DATABASES entry to replace the ENGINE for your connection.
enocom commentedon Mar 6, 2024
Thanks, @rcleveng -- that's a nice approach.
In effect it's a variation of #214 (comment).
It gets you IAM authentication without the Connector. Meanwhile, we're working on improving Django support in the near term.
enocom commentedon Mar 7, 2024
Adding the code sample from the link above for posterity:
JoshTanke commentedon Mar 24, 2024
@enocom Your comments on this thread and others has been really helpful! I'm wondering if you might have any ideas on this related issue I'm hitting. I've been trying to get the IAM auth approach to work with Django on my local machine but the connection keeps timing out. AFAICT my psycopg2.connect database url matches the format in this other thead:
dbname={db_name} client_encoding=UTF8 user={sa}@{project}.iam host={public_ip} port=5432 password={token}. I have public + private IPs enabled and also have "Require trusted client certificates" turned on in the SSL settings. My best guess is the SSL setting might be the issue - do I need to create an SSL client certificate and include it in my DatabaseWrapper somehow, or does the IAM auth flow do that for me somehow?jackwotherspoon commentedon Mar 24, 2024
@JoshTanke Thanks for the comment on this issue. Do you mind creating a new issue on the repo with the error you are seeing? That way we can help debug your problem while keeping this thread clean 😄
Once you open a new issue with your stacktrace I'll get a better grasp of what is going on. However, yes if you have "Require trusted client certificates" turned on and are not creating a database connection using SSL with client certs (mTLS). Which by the sample above and database URL you provided you are not then I would expect it to error. You would need to download your certificates and add their path to your psycopg2 database url (this SO post does a good job explaining the psycopg2 config)
Depending on your use-case and especially for Private IP connections you may be fine with turning the setting to "Allow only SSL connections". This would allow you to update your database url with
sslmode=requireto:bname={db_name} client_encoding=UTF8 user={sa}@{project}.iam host={public_ip} port=5432 password={token} sslmode=requireHowever, this still isn't ideal for Public IP connections which we are working on by adding support for psycopg2 to the Cloud SQL Python Connector soon (will be worked on this quarter) which will then also unblock Django for postgres and give users a way of using Django + Cloud SQL for "Require trusted client certificates" without having to manage certificates yourself.
jjaikumar-xyla commentedon Feb 22, 2025
Is support for psycopg2 still on the roadmap or is there a working code snippet? Would really love to use the Python Connector with my Django+Postgres via psycopg2 production application.
jackwotherspoon commentedon Feb 22, 2025
@jjaikumar-xyla psycopg2 support is in the backlog but not currently prioritized.
I have been playing around a bit in my spare time with trying to get a PoC but still is a WIP.
You can use Cloud SQL with psycopg2 + Django today via the workaround solution in the comments above in this thread:
Let me know if this works for you 😄. I will work on trying to get the team to prioritize psycopg2 support
jjaikumar-xyla commentedon Feb 22, 2025
Nice thanks for the update. My understanding if I'm reading the code correctly is that this approach doesn't use the proxy at all though right? So I would be in charge of implementing credential rotation for a long running service.
Also since this isn't using the proxy, I still need to have my production network listed as an authorized network for the Cloud SQL instance correct?
jackwotherspoon commentedon Feb 23, 2025
@jjaikumar-xyla Correct, it does not use the Proxy so if you are using public IP you would need to update authorized networks. I would recommend the solution above for private networks, if you are using a public IP Cloud SQL connection then Proxy is still your best alternative until we add psycopg2 support to the Python Connector.
jjaikumar-xyla commentedon Feb 23, 2025
Got it, super clear, thank you very much for the info! Would love to see psycopg2 support come through at some point.
foxojs commentedon Dec 12, 2025
Hi, came across this whilst attempting to migrate my django set up in cloud run from password based auth, to IAM, but struggling - I wondered if there is an approach that allows django to connect to postgres db yet via IAM user as was struggling to find anything? (I'm using postGIS)
rcleveng commentedon Dec 14, 2025
I wrote up
https://github.com/rcleveng/django_gcp_iam_auth for that use case. I'm also close to releasing a new repo that can let you use iam auth with both sqlalchemy and psycopg3 pools (requires the new features in psycopg-pool v3.3 that came out December first, have been waiting for the callable support for conning)
Note - that's not an official google project
s-paquette commentedon Feb 27, 2026
We could really use an update on this for MySQL. Starting January next year MySQL 8.0 moves to paid extended support, but 8.4 still isn't supported for Django on App Engine Flex (or Cloud Run). The issue comes down to a lack of support for mysqlclient or mysql-connector-python by the CloudSQL Python Connector, which are what Django supports as engines. Django doesn't appear to have a roadmap or plan to support PyMySQL. There isn't at the moment any documented way to use Django's supported backends with the CloudSQL Python Connector, nor to use PyMySQL in conjunction with Dajngo and the CloudSQL Python Connector.
An added wrinkle here is the CloudSQL Auth Proxy which is built in for App Engine Flex and Cloud Run doesn't seem to have been updated to support MySQL 8.4's default auth mechanism over UNIX sockets, and as far as I know Google's position is users running into this issue should move to the Connector. As indicated above, there's not a path for that at the moment for Django/MySQL users of those services.