IssueTracker

You are right: Dev_appserver decides whether to launch the emulator based on the environment variable DATASTORE_EMULATOR_HOST. This is documented on the "Migrating to the Cloud Datastore Emulator" documentation page [1].

[1] https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator#using_dev_appserver_with_the_datastore_emulator

I think you misread the issue.

dev_appserver DOES launch the datastore emulator correctly, but the Python GAE application that runs from dev_appserver DOES NOT communicate with the local emulator! So yes it is running, but it is pointless because the local GAE app continues to point to the production APIs. Once dev_appserver starts the local emulator, the GAE apps should use it automatically.

Per your docs...

"When this environment variable [DATASTORE_EMULATOR_HOST] does not exist, dev_appserver launches a new emulator process."  and...
"You can force dev_appserver to use the Cloud Datastore emulator by setting the '--support_datastore_emulator' flag to true when starting dev_appserver:"  

In either case, the app that dev_appserver runs does not use the local emulator. This is the issue.

Do you refer to the local Datastore, or the Datastore Emulator?

In case of the local Datastore, your are right, dev_appserver does launch it [1]. By contrast, to launch the Datastore Emulator [2] you need to specifically run command:

gcloud beta emulators datastore start [flags]

You mean that "I should not have to set these options if I enable support_datastore_emulator"; however, documentation states that on the Datastore Emulator you neet to either set environment variables manually [3] with:

export DATASTORE_EMULATOR_HOST=localhost:8432
export DATASTORE_PROJECT_ID=my-project-id

or automatically [4] $(gcloud beta emulators datastore env-init).

Documentation does not state you don't need to set these options. In fact, documentation states [5] the following: "After you start the emulator, you need to set environment variables so that your application connects to the emulator instead of your production Datastore mode database. Set these environment variables on the same machine that you use to run your application. "

[1] https://cloud.google.com/appengine/docs/standard/python/tools/using-local-server#browsing_the_local_datastore
[2] https://cloud.google.com/datastore/docs/tools/datastore-emulator#starting_the_emulator
[3] https://cloud.google.com/datastore/docs/tools/datastore-emulator#manually_setting_the_variables
[4] https://cloud.google.com/datastore/docs/tools/datastore-emulator#automatically_setting_the_variables
[5] https://cloud.google.com/datastore/docs/tools/datastore-emulator#setting_environment_variables

Thanks for the quick reply. The dev_appserver.py "local datastore" only works in Python 2.7 apps/libs. According to Google docs, Python 3 apps need to use the Data Store Emulator to support a local datastore.

I will try to explain another way as I am not running the emulator as a separate step but via the dev_appserver.py feature...

The issue involves the interaction of two Google products and a GAE app:
  - Product A = dev_appserver.py (local app server for GAE apps)
  - Product B = Datastore Emulator (which uses a local file as its datastore)
  - App C = My Python 3.7 GAE app

Here is the sequence:

1. Product A (dev_appserver) has a feature where it will run Product B (emulator) as indicated by this Product A setting: --support_datastore_emulator=True

2. Here is the log output from Product A once that option is included:

INFO     2020-04-15 02:56:34,947 devappserver2.py:232] Using Cloud Datastore Emulator.
We are gradually rolling out the emulator as the default datastore implementation of dev_appserver.
If broken, you can temporarily disable it by --support_datastore_emulator=False
Read the documentation: https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator
Help us validate that the feature is ready by taking this survey: https://goo.gl/forms/UArIcs8K9CUSCm733
Report issues at: https://issuetracker.google.com/issues/new?component=187272

INFO     2020-04-15 02:56:34,953 devappserver2.py:289] Skipping SDK update check.
INFO     2020-04-15 02:56:35,031 datastore_emulator.py:155] Starting Cloud Datastore emulator at: http://localhost:12044
INFO     2020-04-15 02:56:36,582 datastore_emulator.py:161] Cloud Datastore emulator responded after 1.551295 seconds

3. Thus, Product A has launched Product B and it selected a host and port number for Product B automatically (localhost:12044)

4. Next, Product A launches App C (my Python 3.7 GAE app called qb-sync-service) and here is the startup entry for that:

INFO     2020-04-15 02:56:52,310 dispatcher.py:267] Starting module "qb-sync-service" running at: http://localhost:8080

5. App C (qb-sync-service) then uses the Cloud Datastore Python library to access datastore features. The datastore calls *should* go to Product B since dev_appserver launched it and the app. This is what it is broken. In this step, all datastore calls from App C still go to production Google APIs.

Since dev_appserver launched them both, it should handle the necessary env setup to have all Datastore lib calls go to the emulator. The work around I listed was how I 'tricked/forced' App C to use the local emulator. I should not have to force it like this. Dev_appserver should handle the env setup for App C using the host and port number it picked when it launched Product B.

What is the exact command you have run to have product A launch product B, and then C? Is this one single command or two commands: one to launch product A, who launches product B, and the a second command specifically for product C? How did you start product C, i.e. your app?

Assuming product B runs as expected, so Datastore Emulator is now started. Proper use of this, now running, Datastore Emulator requires environment variables to be set: "After you start the emulator, you need to set environment variables so that your application connects to the emulator instead of your production Datastore mode database". In other words, the environment variables are needed by your application to run properly, and then use the local Datastore Emulator as expected, by contrast to the Datastore service in the Cloud.

You expect that these environment variables be set while dev_appserver starts. This is obviously not the case in practice, and these variables need to be set as a separate, required step, just as documentation clearly spells out.

There is only one command:

/Users/Robert/PycharmProjects/abc-sync/venv/bin/python /Users/Robert/Documents/google-cloud-sdk/bin/dev_appserver.py --application=XYZ --log_level=info --max_module_instances=qb-sync-service:1 --storage_path=/Users/Robert/datastore/ --support_datastore_emulator=True --env_var=GOOGLE_APPLICATION_CREDENTIALS=xyz.json qb_sync/service.yaml

Per google docs (https://cloud.google.com/appengine/docs/standard/python3/testing-and-deploying-your-app#local-dev-server)

"The Google Cloud SDK includes a local development server named dev_appserver that you can run locally to simulate your application running in production App Engine. This development server partially simulates the environment in which your application runs."

dev_appserver launches everything else necessary to simulate a GAE prod environment locally. The issue is a bug on dev_appserver and should be routed to that product team. There is no issue with the Datastore Emulator itself.  

Engineering has been made aware of your feature request, and will address it in due course. No estimated time to implementation has been set. Meanwhile, you may follow developments in this thread.