Datastore emulator uses a ton of memory if your database is large [127839125]

Assigned

Bug

Status Update

No update yet.

Description

be...@khanacademy.org

created issue #1

Mar 7, 2019 11:44PM

We're testing out the new cloud datastore emulator for use on our dev servers. Sadly, we've run into a problem: with a large datastore dump (our real one is likely to be like 2GB but for now I'm testing on one that's 846MB) the emulator uses unacceptably much memory (like 3.5GB so far, even before it really does much). It also takes a while to start up -- devappserver actually times out after 10s and thinks it has failed! Is there any hope for us to get this memory usage lower other than by not storing so much data? The old stub often worked with 1GB or less (even on a 2.5GB sqlite file), I suspect since it used sqlite to access the data on-disk rather than loading it all into memory, and we'd love to get back down near there.

Comments

gs...@google.com <gs...@google.com> #2Mar 9, 2019 08:50PM

Assigned to gc...@google.com.

[Comment deleted]

gs...@google.com <gs...@google.com> #3Mar 9, 2019 08:55PM

Would you be able to provide an example app which produces an error for which gcloud doesn't report, but appcfg.py does?

be...@khanacademy.org <be...@khanacademy.org> #4Mar 12, 2019 08:58PM

In this case it is not app specific, The 400 error is due to quota of the application versions being over the limit which is 10 (but for this specific project it has been increased to 60). So any project with more than 10 versions will experience this.

ju...@gmail.com <ju...@gmail.com> #5Apr 4, 2019 03:23PM

[Comment deleted]

be...@khanacademy.org <be...@khanacademy.org> #6Apr 4, 2019 06:12PM

I have forwarded this request/report to the engineering team. We will update this issue with any progress updates and a resolution.

Issue 127839125