Document that this project is in maintenance mode?

[snarfed]

From googleapis/google-cloud-python#10566 (comment) :

The current owners of the Datastore libraries team let me know that NDB is not currently being developed, but things may change in the future.

OK! Fair enough. Maybe consider documenting that in the README and on https://googleapis.dev/python/python-ndb/latest/ ? And also whether it's maintained, and just not actively adding features, or totally unmaintained?

(Thank you for all of your work up until this point, btw!)

[meredithslota]

Hi @snarfed — thanks for filing this issue. Agreed that projects should clearly communicate their level of support. We're doing our planning for the next couple of quarters now and will follow-up.

[jacobg]

I'm sad to read that NDB won't be actively updated moving forwards. NDB is a highly productive library with useful higher level abstractions, such as typed data models, conjunctive/disjunctive queries, etc. Whereas the Python Datastore library is a lower level library that requires a lot more app code for parallel queries, in-memory filtering/merging, type validation, etc. Both have their use cases.

Even if NDB is pretty mature and doesn't need major new features, we still need its dependencies to be actively bumped in order for it to keep up with the latest ecosystem.

[jacobg]

I briefly took a look at what it would take to bump dependencies, i.e., edit setup.py:

        "google-cloud-datastore >= 2.7.0, < 3.0.0",
        "pymemcache",
        "redis",
        "pytz"

It's not quite a trivial upgrade. The whole way the google-cloud-datastore generates code stubs changed:
https://github.com/googleapis/python-datastore/blob/main/UPGRADING.md

There's a lot of calls in the ndb code directly to proto and grpc functions that has totally changed. For me, I'd have to delve into how it all works, which I haven't dealt with before. There'd probably be value in making a community effort, so it's not just every individual doing this privately.

Has anyone attempted this?

[jacobg]

I went ahead, and bumped to google-cloud-datastore 2 on a forked repo here:
https://github.com/jacobg/python-ndb/tree/datastore-v2

Here are the commit notes:

[Release 2.0.0:]
Bump to google-cloud-datastore v2;
Drop Python 2.7 and 3.6 (match versions supported by GAE/GCF runtimes);
Default to Python 3.9

All unit and systems tests pass on my dev computer. I've also updated my own business applications to use it, and there are no known issues.

Here are a few extra thoughts:

1. It would be great if Google were to merge and take on maintaining this v2 branch. It could even be a non-main v2 branch on this repo, as long as they merge future commits into it so that it stays up to date. Or if anyone has suggestions to make this branch a community repo so that the responsibility of maintaining it does not fall upon me.
2. It would great to be able to enable CI on GitHub to run all tests for this fork. It seems that Google has their own internal build system, and they just clone the source code to GitHub without using GitHub for CI.
3. Cloud Datastore recently released native support for !=, IN, and NOT_IN query filters for Firestore in Datastore mode (legacy Datastore not supported). It would be great to take advantage of that NDB to avoid multiple queries where possible. -- https://cloud.google.com/datastore/docs/release-notes#June_17_2022

[snarfed]

@jacobg oh wow, awesome, thank you for doing this! I tried it out and ran the tests for a handful of my projects, ~2k total, and ran a couple of the apps locally and poked at them interactively, and everything passed and seems fine.

Re your extra thoughts, fully agreed with 1 and 3. For 2, I assume you mean the ndb unit tests? Looking at an example PR eg #778, doesn't look like they're run automatically in CI. Maybe try setting up and running them locally?

[jacobg]

@snarfed With regards to CI, what I means is we should configure Travis CI or something like that so that unit tests automatically run when a commit is made. And shows badges on the README. It does seem Google has some of its own build hooks in this repo, but not sure if we would use that for an independent repo, and it doesn't seem they run unit tests.