Occasionally we see errors that lead to the termination of our app engine process. Examples include:

* "Exceeded soft private memory limit"
* "While handling this request, the process that handled this request was found to be using too much memory and was terminated."
* "Process terminated because the request deadline was exceeded."

In MOST cases, we receive no logging output at all from our app when any of these happen. In case #2 ("too much memory") I sometimes do see our own app logs prior to this warning, but other times I see nothing at all logged for a request other than this warning. In the other 2 cases, I believe we are nearly always missing app logs.

As you can imagine, this makes things *very* hard to debug since we do not have much context for what the app was actually doing that lead to these error conditions. Ideally, we'd like if all application logs would be flushed "to disk" whenever one of these errors occurred. If this is not feasible then our second choice would be to expose a "flush" function that would explicitly flush all messages logged thus far (in that handler) so that they are guaranteed to show up in the log console. It appears that python has something similar/identical to this?
https://developers.google.com/appengine/docs/python/logs/functions

It also appears that there is already a background flusher in Go - perhaps a public interface could be exposed to this? https://github.com/golang/appengine/blob/master/internal/api.go#490-491

However I can't explain why this background flusher (which appears to run every 1 second) is not solving the problem for us. I doubt that more than a minority of our dying requests fail to last at least 1 second before being killed by the app engine container.

Lastly, I did find another old issue that seemed related - apologies if this is a dupe: https://code.google.com/p/googleappengine/issues/detail?id=7973

The other issue seemed to be somewhat related to the submitter's specific app having memory problems so I though it would be reasonable to submit a new issue for the general need to cover log messages in the face of fatal errors.

Thanks as always!

Issue 35897517

Links (11)

Issue metadata