Status Update
No update yet.
Comments
ch...@pushbullet.com
import (
"time"
"
basepb "
logpb "
)
// based on
func LogLines(c appengine.Context, lines []string) {
logLines := []*logpb.UserAppLogLine{}
var level int64 = 1
for i := range lines {
logLines = append(logLines, &logpb.UserAppLogLine{
TimestampUsec: proto.Int64(time.Now().UnixNano() / 1e3),
Level: &level,
Message: &lines[index],
})
}
buf, _ := proto.Marshal(&logpb.UserAppLogGroup{
LogLine: logLines,
})
req := &logpb.FlushRequest{
Logs: buf,
}
res := &basepb.VoidProto{}
c.Call("logservice", "Flush", req, res, nil)
return
}
gm...@google.com
I'm guessing that the app is crashing (and a new instance is restarted) which is preventing the log from getting flushed when the catastrophic error occurs. Having a simple app that reliably reproduces this scenario will help us identify and hopefully solve the problem.
Thanks!
ch...@pushbullet.com
Support case:
Filed as feature request #15394190
Reproduction test case:
ch...@pushbullet.com
gm...@google.com
I deployed the application and visited the app from a browser.
It filled the log with 20 lines each time and the logs were visible from the App Engine console.
If this is not what you expected for the method of testing, please provide detailed instructions on how to duplicate the problem.
Thanks.
mi...@docketalarm.com
gm...@google.com
sa...@google.com
Hi,
Does this issue still persist?
Any additional information you provide will help us better understand and resolve the problem.Thank you for your trust and continued support to improve Google Cloud Platform products. In case you want to report a new issue, please do not hesitate to create a new
ia...@gmail.com
is still occurring. If I were you I'd just go ahead and close this ancient
issue!
sent from my phone...
On Mon, Jun 26, 2023, 7:55 AM <buganizer-system@google.com> wrote:
th...@gmail.com
I'm not the original poster, but I can confirm that this issue still exists. When an App Engine instance/process is killed, the app logs for that instance are dropped and are not visible via Cloud Logging.
sa...@google.com
Hello,
Thank you for reaching out to us with your request.
We have duly noted your feedback and will thoroughly validate it. While we cannot provide an estimated time of implementation or guarantee the fulfillment of the issues, please be assured that your input is highly valued. Your feedback enables us to enhance our products and services.
We appreciate your continued trust and support in our Google Cloud Platform products. Should you encounter any further issues or wish to report a new concern, please feel free to create a new thread on the
Description
* "Exceeded soft private memory limit"
* "While handling this request, the process that handled this request was found to be using too much memory and was terminated."
* "Process terminated because the request deadline was exceeded."
In MOST cases, we receive no logging output at all from our app when any of these happen. In case #2 ("too much memory") I sometimes do see our own app logs prior to this warning, but other times I see nothing at all logged for a request other than this warning. In the other 2 cases, I believe we are nearly always missing app logs.
As you can imagine, this makes things *very* hard to debug since we do not have much context for what the app was actually doing that lead to these error conditions. Ideally, we'd like if all application logs would be flushed "to disk" whenever one of these errors occurred. If this is not feasible then our second choice would be to expose a "flush" function that would explicitly flush all messages logged thus far (in that handler) so that they are guaranteed to show up in the log console. It appears that python has something similar/identical to this?
It also appears that there is already a background flusher in Go - perhaps a public interface could be exposed to this?
However I can't explain why this background flusher (which appears to run every 1 second) is not solving the problem for us. I doubt that more than a minority of our dying requests fail to last at least 1 second before being killed by the app engine container.
Lastly, I did find another old issue that seemed related - apologies if this is a dupe:
The other issue seemed to be somewhat related to the submitter's specific app having memory problems so I though it would be reasonable to submit a new issue for the general need to cover log messages in the face of fatal errors.
Thanks as always!