Assigned
Status Update
No update yet.
Comments
ga...@gmail.com
(deleted)
si...@google.com
si...@google.com
Hello,
Could you please confirm if the issue is resolved or not ?
ro...@zachkreutzjans.com
Error is not resolved, I have switched to GKE. Would like to figure out what is wrong with Vertex, because that would be the ideal platform.
ro...@zachkreutzjans.com
Hi, any updates?
si...@google.com
Hello,
To troubleshoot the issue further, I have created a private ticket to provide some information about the issue (for which you should have received a notification). Please provide requested information there. Don't put any personal information, including project identifiers in this public ticket.
Description
I am receiving a timeout from my vertex ai custom container model. Short predictions still work. The long prediction error happens when testing using both nodejs and python prediction client. The funny thing is, the model still completes, I just get no prediction response. First I was getting a "4 DEADLINE EXCEEDED". I fixed that by specifying a higher timeout in call options. Now I get a "Error: 14 UNAVAILABLE: 502:Bad Gateway" for long operations.
I know I can make them longrunning, but I would rather not make model changes just because some predictions run long, plus there seems to be no documentation saying it cannot run a long time.
What you expected to happen:
I expected the model to run to completion, and not timeout on long running predictions.
Steps to reproduce:
Create fastapi/uvicorn custom model. Test from nodejs or python client. You will get a grpc error.
Other information (workarounds you have tried, documentation consulted, etc):
Remember the moel still works, it has a step that uploads to GCS and that file exists. Plus I get "POST /predict HTTP/1.1" 200 OK" in endpoint logs.