Status Update
Comments
pr...@google.com <pr...@google.com>
wh...@google.com <wh...@google.com> #2
This issue seems to be caused by a reinstallation of nvidia drivers when we are restarting VM notebooks.
We are working on a fix, in the meantime, you can use Ubuntu images as a temporary workaround.
yu...@google.com <yu...@google.com> #3
@Hoa, @Hanchao: do you know if this is in any way related to
wh...@google.com <wh...@google.com> #4
Hi Yuxuan, this is the same bug as:
and it doesn't seem related to
wh...@google.com <wh...@google.com> #5
We just released new images with the fix. The issue should be resolved in all TF and PyTorch GPU Debian-10 images.
wh...@google.com <wh...@google.com> #6
Please update to the latest image if you are encountering this issue.
wh...@google.com <wh...@google.com> #7
Steps to upgrade
Step 1: Go to the Notebooks UI.
Step 2: Click on your instance's name.
Step 3: Press the "upgrade" button at the top right corner.
ha...@gmail.com <ha...@gmail.com> #9
al...@gmail.com <al...@gmail.com> #10
wh...@google.com <wh...@google.com> #11
If you spin up the VMs using our AI Platform Notebooks Dashboard, we have an "upgrade" button that will upgrade the existing VM to the latest version. In addition, we also have an option that will keep your environment automatically up to date.
im...@gmail.com <im...@gmail.com> #12
Hi, I used ubuntu image currently and this didn't happened for Ubuntu image, thanks for your solution!
Description
This issue happened when I restarted my cloud notebook server today. Can be reproduced for notebook VM in us-east1-c using the steps below:
Create a Google Cloud Notebook server with Tensorflow or Pytorch and GPU
After starting the server, open the python console:
CUDA device is available until now.
nvidia-smi
command works fine.This issue can also be reproduced by using TensorFlow. How to fix this kind of case?