IssueTracker

This will create a public issue which anybody can view and comment on.

Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.

Problem you have encountered:

At first, we used custom NVidia-based docker image nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu22.04 as a base and 4x Tesla T4 GPU for Pytorch training. It worked correctly - nvidia-smi worked, and we were able to move tensors to CUDA.

Since pytorch is now installing CUDA packages as a dependency, we moved to raw ubuntu22.04 to limit the image size. We tested the image on VM with T4 GPU to make sure it works (obviously, we had to pass --gpus=all to enable GPUs in the container). Then we replaced our nvidia-based image with this new image and run vertex training.

What you expected to happen:

We expected the same results as with the nvidia-based image and the same results as we observed on manually deployed VM with T4. Pytorch library should be able to move tensors to CUDA. Instead we got the generic error:

RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

We connected to the working machine through SSH and noticed that nvidia-smi command is not present. This leads us to believe that either the drivers were not installed on the VM that is running our training or that they were not made visible to the container (e.g. with --gpus=all flag in docker run.

Steps to reproduce:

Create a new docker image and install torch - preferable on a machine with amd64 arch and GPU to make sure that it works. Apply:

FROM --platform=linux/amd64 ubuntu:22.04

ENV \
    PYTHONUNBUFFERED=1 \
    DEBIAN_FRONTEND=noninteractive \
    TZ=Europe/Warsaw

RUN apt-get -qq update \
    && apt-get -qq -y install \
    curl \
    jq \
    git \
    build-essential \
    gcc \
    libb64-dev \
    software-properties-common \
    && apt-get install python3 python3-pip -y \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get -qq -y clean

RUN umask 0000 \
    && pip install --upgrade pip setuptools

RUN umask 0000 && pip install torch==1.13.1 torchvision torchaudio

Other information (workarounds you have tried, documentation consulted, etc):

As I mentioned, basing the image on nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu22.04 removes the issue, but the image is significantly larger because it then includes the CUDA twice.