IssueTracker

We're trying to test out the new datastore emulator, and in particular are attempting to migrate our data from the old dev_appserver stub (per https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator).  Sadly, we've run into a rather strange problem, where, on certain input data, the conversion crashes because of some mismatch with dev~ and s~ keys.

The conditions for the input data are rather strange: I've been creating files to test import by just taking the first n entities in the dump we use for our dev servers.  It seems to crash any time I include at least the first 180 entities (about 2.9M of data in sqlite form).  Interestingly, though, it's not like it's the 180th entity that's the problem: the entity it crashes on varies depending on how many entities I include in the dump, and is often an entity included even in smaller working dumps.  So it seems to be something about the size of the import that causes things to crash.

Here's the command I'm running, and the error I get:
---
$ dev_appserver.py --application=khan-academy --datastore_path=datastore/bad.sqlite --support_datastore_emulator=true ./app.yaml
INFO     2019-02-28 01:30:56,643 devappserver2.py:224] Using Cloud Datastore Emulator.
We are gradually rolling out the emulator as the default datastore implementation of dev_appserver.
If broken, you can temporarily disable it by --support_datastore_emulator=False
Read the documentation: https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator
Help us validate that the feature is ready by taking this survey: https://goo.gl/forms/UArIcs8K9CUSCm733
Report issues at: https://issuetracker.google.com/issues/new?component=187272

WARNING  2019-02-28 01:30:56,655 appinfo.py:1656] pytz version "2016.4" is deprecated, use one of: "2017.3"
WARNING  2019-02-28 01:30:56,656 appinfo.py:1656] setuptools version "0.6c11" is deprecated, use one of: "36.6.0"
WARNING  2019-02-28 01:30:56,657 appinfo.py:1656] webapp2 version "2.5.1" is deprecated, use one of: "2.3", "2.5.2"
INFO     2019-02-28 01:30:56,791 devappserver2.py:278] Skipping SDK update check.
INFO     2019-02-28 01:30:56,792 api_server.py:508] SQLite stub data has been renamed to datastore/test.sqlite.sqlitestub
INFO     2019-02-28 01:30:56,894 datastore_emulator.py:155] Starting Cloud Datastore emulator at: http://localhost:15761
INFO     2019-02-28 01:30:57,813 datastore_emulator.py:161] Cloud Datastore emulator responded after 0.919146 seconds
INFO     2019-02-28 01:30:57,813 api_server.py:499] Converting datastore_sqlite_stub data in datastore/test.sqlite.sqlitestub to Cloud Datastore emulator data in datastore/test.sqlite.
INFO     2019-02-28 01:30:58,112 datastore_converter.py:101] Fetched 180 entities from datastore/test.sqlite.sqlitestub
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/benkraft/khan/devtools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 501, in _launch
    app_id, sqlite_data_renamed, emulator_host)
  File "/home/benkraft/khan/devtools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/datastore_converter.py", line 105, in convert_sqlite_data_to_emulator
    datastore.Put(entities)
  File "/home/benkraft/khan/devtools/google-cloud-sdk/platform/google_appengine/google/appengine/api/datastore.py", line 622, in Put
    return PutAsync(entities, **kwargs).get_result()
  File "/home/benkraft/khan/devtools/google-cloud-sdk/platform/google_appengine/google/appengine/datastore/datastore_rpc.py", line 927, in get_result
    results = self.__extra_hook(results)
  File "/home/benkraft/khan/devtools/google-cloud-sdk/platform/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1485, in sort_result_index_pairs
    results = extra_hook(results)
  File "/home/benkraft/khan/devtools/google-cloud-sdk/platform/google_appengine/google/appengine/api/datastore.py", line 587, in local_extra_hook
    assert not entity._Entity__key.has_id_or_name()
AssertionError
---
Whereas, if I use the dump with only the first 179 entities, it works fine (continuing to say "Datastore conversion complete").

Adding some extra print statements just above that assertion I can get that entity._Entity__key._Key__reference (the key of the entity we sent, I think) is:
app: "dev~khan-academy"
path <
  Element {
    type: "ArticleLog"
    id: 0x14d6c000000000
  }
>
and key._Key__reference (the returned key, I think) is:
app: "s~khan-academy"
path <
  Element {
    type: "ArticleLog"
    id: 0x14d6c000000000
  }
>
As you can see, the only difference is the app ID: we send the entity with dev~ but get back s~.  Interestingly, this is the first entity in the alphabetically-first kind in the dump, so it is included in both the good and bad dumps.

I don't want to post the actual datastore dumps here -- since I wasn't able to find a truly small example they do probably contain some data that shouldn't be public -- but I can share them with someone at Google if that's helpful, just let me know how.  Or let me know if there is other useful debugging information I can give you.