Localhost GAE app (Python 3) can't connect with a localhost Datastore Emulator (DefaultCredentialsError) [123892383]

Assigned

Bug

Status Update

No update yet.

Description

ma...@gmail.com

created issue #1

Feb 4, 2019 09:41PM

Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.

Problem you have encountered:

I'm receiving the DefaultCredentialsError when my GAE Python 3 web app on localhost wants to connect on localhost Datastore emulator (on the same machine).

What you expected to happen:

That the app would be able to connect with the Datastore Emulator without needing any credentials.

Steps to reproduce:

Run this docker example:

https://github.com/ramuta/docker-gae-py3-datastore (run this command in the Terminal: docker-compose up --build).

There's a version on the master branch with Datastore Emulator running in the SAME docker container as the GAE app. On the "datastore-separate-container" branch you can run datastore and GAE app in SEPARATE containers. Both options reproduce the same error: DefaultCredentialsError.

Other information (workarounds you have tried, documentation consulted, etc):

I've checked all the docs and parts of the source code. Nothing works. Also asked on StackOverflow, but no luck either:

https://stackoverflow.com/questions/54507222/localhost-how-to-get-credentials-to-connect-gae-python-3-app-and-datastore-emul

Any idea how to solve this?

Kind regards,
Matej Ramuta

Comments

gs...@google.com <gs...@google.com> #2Feb 4, 2019 11:33PM

Assigned to gs...@google.com.

After you start the emulator, you need to set environment variables so that your application connects to the emulator instead of the production Datastore mode environment. Set these environment variables on the same machine that you use to run your application. How did you set these environment variables? A detailed list would be appreciated. You may refer to the "Running the Datastore mode Emulator" documentation page. [1]

[1]

https://cloud.google.com/datastore/docs/tools/datastore-emulator#setting_environment_variables

ma...@gmail.com <ma...@gmail.com> #3Feb 4, 2019 11:40PM

Hi, as you can see in the Dockerfile, the env vars are set:

DATASTORE_EMULATOR_HOST

0.0.0.0:24000
DATASTORE_PROJECT_ID test

gs...@google.com <gs...@google.com> #4Feb 4, 2019 11:43PM

You need to set the environment variables each time you start the emulator. The environment variables depend on dynamically assigned port numbers that could change when you restart the emulator. How did you account for the fact that dynamically assigned port numbers that could change? Your host address looks fixed.

ma...@gmail.com <ma...@gmail.com> #5Feb 5, 2019 12:13AM

I use this command:

gcloud beta emulators datastore start --host-port=

0.0.0.0:24000

The --host-port flag sets the fixed port, so the ports are NOT dynamically assigned.

That's how I can know which port will be used.

Plase see this file:

https://github.com/ramuta/docker-gae-py3-datastore/blob/datastore-separate-container/docker-compose.yml

ma...@gmail.com <ma...@gmail.com> #6Feb 5, 2019 12:15AM

The port is correct and works. If I visit

0.0.0.0:24000 in my browser I get the 200 ok response, so the emulator and the port are okay.

The problem is that the web app can't find the Datastore Emulator.

ma...@gmail.com <ma...@gmail.com> #7Feb 5, 2019 02:38PM

The problem could be connected to the project/app ID.

In the env vars I set the project ID as "test".

If I open the API server in browser I get the following response: {app_id: dev~None, rtok: '0'}

Which means the app ID is not set.

So I add the "-A test" flag to the dev_appserver.py command.

If I visit the API server again in my browser, I see this response: {app_id: dev~test, rtok: '0'}

So the ID is prepended with "dev~" which probably breaks the connection with the Datastore emulator.

See example on the master branch:

https://github.com/ramuta/docker-gae-py3-datastore (read README.md to build properly).

gs...@google.com <gs...@google.com> #8Feb 5, 2019 11:38PM

If your application and the emulator run on the same machine, you can set the environment variables automatically; run env-init using command substitution: $(gcloud beta emulators datastore env-init) . Does this recommended approach work for you? It is worthwhile trying, even if later on you decide upon implementing a different solution. You may refer to "Running the Datastore mode Emulator". [1]

[1]

https://cloud.google.com/datastore/docs/tools/datastore-emulator#automatically_setting_the_variables

ma...@gmail.com <ma...@gmail.com> #9Feb 5, 2019 11:44PM

I've tried this, doesn't help. I get the same credentials error in the end.

Can you try to run this docker setup yourself? Otherwise you cannot really see what's going on.

gs...@google.com <gs...@google.com> #10Feb 5, 2019 11:48PM

The default value is localhost:8081. It is advisable to try this value first, as a test. You may refer to "gcloud beta emulators datastore start" documentation page. [1]

You do not mention how you run dev_appserver.py. You need to set:

--support_datastore_emulator=yes|no
Use the Cloud Datastore Emulator (beta) for local datastore emulation.

As well as:

--datastore_emulator_port=...
The port number to use for Cloud Datastore Emulator (beta).

More dev_appserver.py information is to be found on the "Python 3.7 Local Development Server Options" page. [2]

[1]

https://cloud.google.com/sdk/gcloud/reference/beta/emulators/datastore/start
[2]

https://cloud.google.com/appengine/docs/standard/python3/tools/local-devserver-command

Message last modified on Feb 5, 2019 11:58PM

ma...@gmail.com <ma...@gmail.com> #11Feb 5, 2019 11:53PM

I also did this. The default port value and the fixed port value. I got the credentials error in both cases. Please take 5 minutes to run the docker setup.

I created this GitHub project so that anyone could easily set up a new GAE project with a Datastore Emulator. The dockerized environment is the best way to achieve this. I'm spending my free time to work on this open sorce project (which will also benefit Google), so please take 5 minutes to try to run it yourself.

da...@gmail.com <da...@gmail.com> #12Feb 6, 2019 03:59PM

Tried to run Matejs code today and got the same error (DefaultCredentialsError).

Message last modified on Feb 7, 2019 01:37PM

gs...@google.com <gs...@google.com> #13Feb 8, 2019 09:29PM

Reassigned to gc...@google.com.

This issue has been made known to Engineering, who will address it in due course. No estimated time to resolution has been set. Meanwhile, you may follow developments in this thread.