Skip to content

Having trouble querying by KeyProperty #1684

Closed
Closed
Having trouble querying by KeyProperty#1684
Assignees
dhermes
Labels
api: datastoreIssues related to the Datastore API.

Description

@nlokare
nlokare
opened on Mar 30, 2016

Hi,

I am having trouble querying entities in NDB from AppEngine by KeyProperty for entities created with a datastore.Key from the gcloud.datastore library. When I log the KeyProperty of the saved entity on AppEngine, I see a return value of Key('Company', '273', app='random_project_id'). Because I am attempting to query by Key Literal on AppEngine, it looks like NDB does not recognize the above Key with a query filter such as Entity.query(Entity.key_property == ndb.Key('Company', '273').fetch(). This query should return an iterator with various entities.

Is this a potential bug with gcloud or with AppEngine? @lexsf @vliang63

Activity

  1. dhermes
    added
    api: datastoreIssues related to the Datastore API.
    on Mar 30, 2016

  2. dhermes commented on Mar 30, 2016

    @dhermes
    dhermes
    on Mar 30, 2016
    Contributor

    @pcostell WDYT? Seems it may be related to #1639

  3. dhermes commented on Mar 31, 2016

    @dhermes
    dhermes
    on Mar 31, 2016
    Contributor

    @nlokare Can we try to throw together a reproducible example? You're inserting the entities from gcloud-python and then the GAE / ndb queries are failing, yes?

  4. nlokare commented on Mar 31, 2016

    @nlokare
    nlokare
    on Mar 31, 2016
    Author

    @dhermes Exactly.

    On GCE

    account_key = datastore.key("Company", account_id)
...
advertiser['client_key'] = account_key
datastore.put(advertiser)

    On GAE

    def return_all_advertisers(self, account_id):
   all_advertisers = Advertiser.query(Advertiser.client_key == ndb.Key('Company', account_id)).order(Advertiser.advertiser_name).fetch()

  5. nlokare commented on Mar 31, 2016

    @nlokare
    nlokare
    on Mar 31, 2016
    Author

    @dhermes Just to clarify, my queries work on GAE when developing locally because we seed our datastore directly from GAE. Alternatively, in production/staging we are populating our datastore from GCE. Our GAE application is essentially only reading from the datastore.

  6. pcostell commented on Mar 31, 2016

    @pcostell
    pcostell
    on Mar 31, 2016
    Contributor

    Sounds like the same issue as #1639. Same as that issue, I think this will be fixed in v1beta3 if we always populate the project in 'gcloud-python'.

    @nlokare Can you try again but with something like this on GCE:

    account_key = datastore.key("Company", account_id, project_id="<your-application-id>")
...
advertiser['client_key'] = account_key

    Where the application contains your project ID but with the extra information (likely s~).

  7. nlokare commented on Mar 31, 2016

    @nlokare
    nlokare
    on Mar 31, 2016
    Author

    @pcostell I will try this. However, if my GAE ndb.Key object doesn't contain the project_id keyword argument, I think that this will still fail, since the creation of the entity from GCE added the extra data to the Key that wasn't implicit in the GAE query. Does that make sense?

    In the pseudo-code above, I initialized the datastore object with our project. I left that out in the code snippet.

  8. pcostell commented on Mar 31, 2016

    @pcostell
    pcostell
    on Mar 31, 2016
    Contributor

    How do you print the key? Can you try printing key.app() directly? ndb.Key's default __repr__ won't print the application ID if it matches the current app.

  9. nlokare commented on Mar 31, 2016

    @nlokare
    nlokare
    on Mar 31, 2016
    Author

    From GAE I print the key doing:

    all_advertisers = Advertiser.query().order(Advertiser.advertiser_name).fetch(15)
advert_tester = all_advertisers[0]
test_key = advert_tester.client_key
logging.info(test_key)

    This is how we discovered that the Entity was saved from GCE with the client_key property as Key('Company', '273', app='random_project_id')

  10. dhermes commented on Mar 31, 2016

    @dhermes
    dhermes
    on Mar 31, 2016
    Contributor

    OK I just confirmed:

    from gcloud import datastore

project = 'MY-PROJ'
client1 = datastore.Client(project=project)
entity1 = datastore.Entity(key=client1.key('Bar', 1))
entity1['foo'] = client1.key('Foo', 1234)
client1.put(entity1)
client2 = datastore.Client(project='s~' + project)
entity2 = datastore.Entity(key=client2.key('Bar', 2))
entity2['foo'] = client2.key('Foo', 1234)
client2.put(entity2)

    and then from the remote API

    $ ~/google-cloud-sdk/platform/google_appengine/remote_api_shell.py MY-PROJ
App Engine remote_api shell
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2]
The db, ndb, users, urlfetch, and memcache modules are imported.
s~MY-PROJ> from google.appengine.ext import ndb
s~MY-PROJ> class Foo(ndb.Model):
...            pass
...
s~MY-PROJ> class Bar(ndb.Model):
...            foo = ndb.KeyProperty(kind='Foo')
...
s~MY-PROJ> q = Bar.query(Bar.foo == ndb.Key(Foo, 1234))
s~MY-PROJ> q.fetch()
[Bar(key=Key('Bar', 2), foo=Key('Foo', 1234))]

  11. nlokare commented on Mar 31, 2016

    @nlokare
    nlokare
    on Mar 31, 2016
    Author

    @dhermes So, adding the s~ before the project when initializing datastore.Client will solve this?

  12. dhermes commented on Mar 31, 2016

    @dhermes
    dhermes
    on Mar 31, 2016
    Contributor

    Yup, that's what @pcostell mentioned above:

    with the extra information (likely s~).

  13. nlokare commented on Mar 31, 2016

    @nlokare
    nlokare
    on Mar 31, 2016
    Author

    @dhermes @pcostell Ok, great. I will try this and report back. Thanks for help!

  14. dhermes commented on Mar 31, 2016

    @dhermes
    dhermes
    on Mar 31, 2016
    Contributor

    You could also use find_true_project but that all is moot in v1beta3.

  15. nlokare commented on Mar 31, 2016

    @nlokare
    nlokare
    on Mar 31, 2016
    Author

    @dhermes @pcostell Reporting back. Looks like this was the fix we needed. Thank you gentlemen.

  16. dhermes commented on Mar 31, 2016

    @dhermes
    dhermes
    on Mar 31, 2016
    Contributor

    Sounds good. Closing this for same reason we closed #1639.

  17. dhermes
    closed this as completedon Mar 31, 2016
  18. JustinBeckwith
    assigned
    dhermes
    on Feb 1, 2021
  19. parthea
    added a commit that references this issue on Nov 24, 2025

    test: fix issue in tests related to json serialization (#1684)

    17e6065
  20. parthea
    added a commit that references this issue on Mar 6, 2026

    chore(deps): bump urllib3 from 1.26.15 to 1.26.18 in /samples/geograp…

    f368b73
  21. parthea
    added a commit that references this issue on Apr 1, 2026

    fix: Guarantee guid thread safety across threads (#1684)

    1225c00
  22. parthea
    added a commit that references this issue on Sep 8, 2026

    docs: migrate code from googleapis/python-bigquery (#18303)

    23f9a74
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: datastoreIssues related to the Datastore API.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions