"gcloud compute ssh" can't be used in parallel [149709703]

WAI

Bug

Status Update

No update yet.

Description

ga...@gmail.com

created issue #1

Feb 18, 2020 11:50AM

for h in stage-agency-admin stage-am-batch-server stage-async; do (echo ok | gcloud compute ssh $h -- echo $h ok; echo $h: $?;)& done

bash# for h in stage-agency-admin stage-am-batch-server stage-async; do (echo ok | gcloud compute ssh $h -- echo $h ok; echo $h: $?;)& done
[1] 27626
[2] 27627
[3] 27630
External IP address was not found; defaulting to using IAP tunneling.
Pseudo-terminal will not be allocated because stdin is not a terminal.
External IP address was not found; defaulting to using IAP tunneling.
Pseudo-terminal will not be allocated because stdin is not a terminal.
External IP address was not found; defaulting to using IAP tunneling.
Pseudo-terminal will not be allocated because stdin is not a terminal.
ERROR: gcloud crashed (OperationalError): disk I/O error
ERROR: gcloud crashed (OperationalError): disk I/O error

If you would like to report this issue, please run the following command:

If you would like to report this issue, please run the following command:
gcloud feedback
gcloud feedback

To check gcloud for common problems, please run the following command:

To check gcloud for common problems, please run the following command:
gcloud info --run-diagnostics
gcloud info --run-diagnostics
ssh_exchange_identification: Connection closed by remote host
ssh_exchange_identification: Connection closed by remote host
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
stage-am-batch-server: 255
stage-agency-admin: 255
stage-async ok
stage-async: 0

I am trying to use "gcloud compute ssh" in wrapper script to enable access for Ansible when we are behind OS Login:

https://www.reddit.com/r/ansible/comments/e9ve5q/ansible_slow_as_a_hell_with_gcp_iap_any_way_to/

when I run more stressful:

for ((i=0; i<10; i+=1)) do for h in stage-agency-admin stage-am-batch-server stage-async stage-sabre-batch-server stage-sabre-rp stage-smart-agent-service stage-tp-batch-server; do (echo ok | gcloud compute ssh --tunnel-through-iap $h -- echo $h ok; echo $h: $?;)& done; done

where is extra warning and errors:

ERROR: gcloud crashed (OperationalError): disk I/O error
WARNING: Could not store access token in cache: disk I/O error
WARNING: Could not store access token in cache: disk I/O error
ERROR: gcloud crashed (DatabaseError): database disk image is malformed
ERROR: gcloud crashed (OperationalError): duplicate column name: id_token
WARNING: Could not store access token in cache: disk I/O error
ERROR: gcloud crashed (OperationalError): disk I/O error
ERROR: gcloud crashed (OperationalError): disk I/O error
ERROR: gcloud crashed (OperationalError): disk I/O error
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: gcloud crashed (OperationalError): disk I/O error
ERROR: gcloud crashed (OperationalError): disk I/O error
ERROR: gcloud crashed (OperationalError): disk I/O error

I run that in WSL.

My college froze Gnome on Linux with latest command.

I assume that local sqlite (or something like that) is in use and it forbids concurrent modifications.

I assume that gcloud MUST NOT be called in parallel.

Comments

pi...@google.com <pi...@google.com> #2Feb 20, 2020 10:31AM

Assigned to pi...@google.com.

Hi,

the

reddit.com website is down, could you post the details?

Thanks, waiting for your answer.

ga...@gmail.com <ga...@gmail.com> #3Feb 20, 2020 12:48PM

Now reddit is up. See also

https://unix.stackexchange.com/questions/545034/with-ansible-is-it-possible-to-connect-connect-to-hosts-that-are-behind-cloud-i

In nutshell: when OS Login is used and public IP addresses are disabled it is possible to SSH into host via `gcloud compute ssh --tunnel-through-iap`.

With a bit of magic it is possible to write wrapper for `gcloud compute ssh --tunnel-through-iap` that acts as `ssh` tool ))

Next step is to run wrappers in parallel (to speedup deploy time by Ansible according to number of forks). When `gcloud compute ssh` is invoked in parallel we frequently experienced errors. It happens seldom even using one call at time but almost 100% when in parallel.

ba...@google.com <ba...@google.com> Feb 28, 2020 09:45AM

Reassigned to gc...@google.com.

al...@google.com <al...@google.com> #4Feb 28, 2020 02:08PM

Hello,

The product engineering team is investigating the issue and is working on a resolution.

Thank you for your patience.

sh...@google.com <sh...@google.com> #5Feb 28, 2020 03:38PM

Hi,

gcloud does not support parallel execution, particually for auth related operations. The errors are mainly due to locks on local sqlite we use for credentials.

A workaround this might be to use gcloud compute ssh --tunnel-through-iap and then issue standard ssh commands from one machine in the subnet. i.e. push the parallelization outside gcloud.

ga...@gmail.com <ga...@gmail.com> #6Mar 1, 2020 07:53AM

Thanks for the conformation! We'll implemented jumping into a subnet with --tunnel-through-iap.

oa...@google.com <oa...@google.com> Aug 19, 2020 01:37PM

Status: Won't Fix (Intended Behavior)