IssueTracker

We have been running all of our CI in Cloud Build for the past year. We have a docker network specified similar to:

networks:
  ournetwork:
    name: cloudbuild
    external: true

This has always worked fine. Suddenly today all of our Cloud Build runs fail with services in our network saying "Connection Refused". There have been no changes to our Cloud Build setup or any recently merged PRs.

For example, we have containers for Mongo. mongo and mongotest containers are definitely attached to the cloud build network which we can verify by checking the output of docker network inspect cloudbuild , but neither hostname mongo or mongotest are resolving, even connecting directly to their IPs.

Sample check we ran:

  - id: "test_mongo_port"
    wait_for:
      - start_mongodb
    name: 'gcr.io/cloud-builders/gcloud'
    allowExitCodes: [ 0 ]
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        apt update && apt install -y netcat jq
        jq -r '.[0].Containers[] | select(.Name == "mongo") | .IPv4Address' /workspace/network.json | cut -d '/' -f1 | xargs -I {} nc -zv {} 27017 && echo "test mongo IP is OK"
        jq -r '.[0].Containers[] | select(.Name == "mongotest") | .IPv4Address' /workspace/network.json | cut -d '/' -f1 | xargs -I {} nc -zv {} 27000 && echo "test mongotest IP is OK"
        cat /workspace/network.json | jq
        i=0
        while [ $i -lt 60 ]; do
          nc -zv mongotest 27000 && echo "mongotest is up" && exit 0
          nc -zv mongocfg1 27017 && echo "mongocfg1 is up" && exit 0
          i=$((i+1))
          sleep 10
        done