Fixed

Bug

Status Update

No update yet.

Description

ne...@omnibond.com

created issue #1

Jul 30, 2020 07:04PM

Problem you have encountered:
After applying latest CentOS 7 updates, the instance is un-bootable.

What you expected to happen:

Steps to reproduce:
Create a CentOS 7 vm
update (sudo yum update)
reboot

Other information (workarounds you have tried, documentation consulted, etc):
!!!! X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID - 00000000 !!!!

ExceptionData - 0000000000000000

RIP - 00000000BF2E6D5C, CS - 0000000000000038, RFLAGS - 0000000000010006

RAX - 00000000BFF67E50, RCX - 000000000000001F, RDX - 00000000BFF39F28

RBX - 00000000BF352FB0, RSP - 00000000BFF39E50, RBP - 00000000000023C0

RSI - 00000000BFF39F28, RDI - 0000000000000068

R8 - 00000000BDB79018, R9 - 000000000000211E, R10 - 00000000BDC59010

R11 - 00000000BFF39E9A, R12 - 00000000BE757068, R13 - 00000000BF339018

R14 - 00000000BEE21018, R15 - 0000000000000068

DS - 0000000000000030, ES - 0000000000000030, FS - 0000000000000030

GS - 0000000000000030, SS - 0000000000000030

CR0 - 0000000080010033, CR2 - 0000000000000000, CR3 - 00000000BF401000

CR4 - 0000000000000668, CR8 - 0000000000000000

DR0 - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000

DR3 - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400

GDTR - 00000000BF3EEA98 0000000000000047, LDTR - 0000000000000000

IDTR - 00000000BEE1F018 0000000000000FFF, TR - 0000000000000000

FXSAVE_STATE - 00000000BFF39AB0

!!!! Find image based on IP(0xBF2E6D5C) /build/work/af60adde42b1d1ad5be2a01e4924bb905248/google3/blaze-out/k8-opt/genfiles/third_party/edk2/ovmf_x64_csm_debug_workspace_dir/ovmf_x64_csm_debug_edk2_files_dir/Build/OvmfX64/DEBUG_CLANG38/X64/OvmfPkg/8254TimerDxe/8254Timer/DEBUG/Timer.dll (ImageBase=00000000BF2E5000, EntryPoint=00000000BF2E6AB5) !!!!

Comments

hi...@google.com <hi...@google.com> Jul 30, 2020 07:42PM

Assigned to hi...@google.com.

bl...@google.com <bl...@google.com> #2Jul 30, 2020 08:06PM

We have shared this with our product and engineering team and will update this issue with more information as it becomes available.

Message last modified on Jul 30, 2020 10:37PM

hi...@google.com <hi...@google.com> #3Jul 30, 2020 08:09PM

Thanks for reporting this issue.

If you are calling our API and when there are multiple "authorized" accounts, we do need to show the list so user can select the one they need to use for that session. Alternatively, you can have your own management system so when a user signs into your account, you won't need to call our API again and can let the user in based n your own management system.

zm...@google.com <zm...@google.com> #4Jul 30, 2020 10:07PM

We believe with reference to the above comment, your query has been answered, hence closing the bug. Please feel free to re-open the issue in the future if desired.

aa...@gmail.com <aa...@gmail.com> #5Jul 31, 2020 01:03AM

It's because we trust google signin system so we want to use google process to verify that user actually is own the email they want to login with. It just for the convenience that the process can infer the email to use and continue the flow without selection screen if we know what email to used beforehand

There are all benefit that google have been allow signin hint before and you just remove it in the new version. Thus making migration problem without equivalence solution

Please reopen and fix it

Screen Shot 2020-07-30 at 8.55.33 PM.png

309 KB

View

Download

[Deleted User] <[Deleted User]> #6Jul 31, 2020 12:27PM

This was notified me

What's changed?

je...@microdinc.com <je...@microdinc.com> #7Jul 31, 2020 06:29PM

Still facing this issue. Once you run yum update the instance is unavailable Running Centos 7

ch...@enviemedia.com <ch...@enviemedia.com> #8Jul 31, 2020 09:23PM

anyone find a way to fix this?

ma...@gmail.com <ma...@gmail.com> #9Jul 31, 2020 09:27PM

I am affected with this as well. Has anyone find a workaround?

Message last modified on Jul 31, 2020 09:45PM

fg...@google.com <fg...@google.com> #10Jul 31, 2020 09:55PM

The workaround for this issue has been posted here [1]

[1]

https://access.redhat.com/solutions/5272311

we...@gmail.com <we...@gmail.com> #11Aug 1, 2020 01:57PM

Does google have a plan to resolve this for affected machines? We can't exactly restore them with a RHEL DVD as proposed by redhat....

fa...@gmail.com <fa...@gmail.com> #12Aug 1, 2020 03:12PM

After 48 hours of this incident and Google has not solved this problem-

we...@gmail.com <we...@gmail.com> #13Aug 1, 2020 03:28PM

How they did not notify users is mind-blowing. Lost 2 full days of work after rebooting this morning. The proposed "solutions" don't work, wasted several hours restoring snapshots, and attempting solutions only to have the system freeze at POST. Completely unacceptable.

lu...@gcp.somosnuvem.com.br <lu...@gcp.somosnuvem.com.br> #14Aug 1, 2020 07:34PM

Enfim, sem saber do problema. Agora estou com um cliente afetado pelo mesmo.

ma...@gmail.com <ma...@gmail.com> #15Aug 1, 2020 10:42PM

It has been a frustrating 2 days...

Message last modified on Aug 1, 2020 10:42PM

se...@gmail.com <se...@gmail.com> #16Aug 1, 2020 10:44PM

Still unsolved...

sa...@etcc.com <sa...@etcc.com> #17Aug 2, 2020 12:16AM

The work around provided does not resolve the issue. I mounted the boot disk to another VM, chroot into it, and downgraded. The same boot error still occurred.

ba...@gmail.com <ba...@gmail.com> Aug 2, 2020 01:12AM

deleted

Restricted

0 B

in...@larsendigital.dk <in...@larsendigital.dk> #18Aug 2, 2020 12:58PM

When is this expected to be resolved?

we...@gmail.com <we...@gmail.com> #19Aug 2, 2020 08:40PM

Any updates? I restored a snapshot but am waiting to be able to update. Also would be nice if it's possible to recover the 15+ hours of work I lost.

ho...@gmail.com <ho...@gmail.com> #20Aug 3, 2020 04:06AM

https://stackoverflow.com/a/63199084/11042591 This answer worked for me

vi...@gmail.com <vi...@gmail.com> #21Aug 3, 2020 05:01AM

In my case I had to change temporarily the affected build's script to exclude shim-x64 from the updates. Any centos-7 image built with this shim affected version shim-x64-15-7.el7_9.x86_64 gets locked and unusable.
When the solution comes up I will revert this change.

na...@gmail.com <na...@gmail.com> #22Aug 3, 2020 08:21AM

This answer worked for me:

https://stackoverflow.com/questions/63187796/unable-to-access-the-vm-after-first-reboot-when-using-startup-script

na...@gmail.com <na...@gmail.com> #23Aug 3, 2020 08:25AM

Although after downgrading the shim, my VM did boot up, but can someone confirm that the shim update containing the bug will not be applied to my VM if I reboot again today.

ma...@gmail.com <ma...@gmail.com> #24Aug 3, 2020 09:39AM

Hey Google, any update on this?

st...@google.com <st...@google.com> Aug 3, 2020 03:56PM

Reassigned to st...@google.com.

st...@google.com <st...@google.com> #25Aug 3, 2020 06:25PM

Updates have been rolled out by Redhat and CentOS. If your instance is booting correctly, you can update to the following shim versions:

RHEL 7: shim-x64-15-8.el7_8.x86_64
RHEL 8: shim-x64-15-15.el8_2.x86_64
CentOS 7: shim-x64-15-8.el7.x86_64
CentOS 8: shim-x64-15-15.el8_2.x86_64

You can see current version with rpm -q shim-x64. Do not update to a lower version than those given above.

If your machine is currently hung at boot, you will need to copy the shim from a working machine. See the instructions in comment #2. Already hung machines will require intervention to boot again.

UPDATE: Previously the package for RHEL 7 had not rolled out. At this time packages are available for all OS versions affected.

UPDATE 2: see comment #29 for fixing already hung machines.

Message last modified on Aug 5, 2020 01:39PM

st...@google.com <st...@google.com> #26Aug 3, 2020 10:54PM

The updated packages from comment #25 are now available for all OS versions.

Message last modified on Aug 3, 2020 10:55PM

fa...@gmail.com <fa...@gmail.com> #27Aug 5, 2020 01:54AM

hello google (st...@google.com), my instance does not turn on. How can I carry out the "yum update shim-x64" update? In comment # 25 you mention that the machines require intervention to boot again.

Excuse me, but your message is very concise- how come it should have an intervention to boot again, what do you mean?

If you can explain the procedure in more detail I would appreciate it.

Thank you!

id...@gmail.com <id...@gmail.com> #28Aug 5, 2020 08:41AM

so desperate, is out there step by step tutorial how to fix this issue?

help please!

st...@google.com <st...@google.com> #29Aug 5, 2020 01:36PM

Instructions for recovering a hung machine:

Backup impacted instance

Before doing anything, snapshot the impacted disk! Mistakes happen.

Example (see gcloud compute disks snapshot --help for all options or use the Cloud Console):

gcloud compute disks snapshot DISK_NAME --zone ZONE --snapshot-names=NAME

Build rescue instance

Create a new instance with the same OS as the damaged disk in the same zone as the damaged disk. An instance from a new image won't have the updates applied and therefore should boot without hanging.

Add the impacted disk as a additional disk to the rescue machine

Stop the impacted instance (the one that hangs at boot).
Edit the impacted instance in the Cloud Console and remove its boot disk
Edit the rescue instance and attach the impacted boot disk as an additional disk.
SSH'd into the rescue machine, mount the EFI boot partition of the impacted instance to /mnt. Assuming you have only attached one disk and these are Google images, you need to mount /dev/sdb1 /mnt. You can use lsblk to determine the actual device name.

Repair the impacted disk

RHEL

cp /boot/efi/EFI/redhat/shimx64.efi /mnt/EFI/redhat/

CentOS

cp /boot/efi/EFI/centos/shimx64.efi /mnt/EFI/centos/

Repair impacted instance

Unmount the disk from rescue instance: umount /mnt
Detach impacted disk from rescue instance.
Attach disk as boot disk to impacted instance
Start impacted instance
Optional: snapshot again. Can't be too careful.
On impacted instance, update to a known working version of the shim: yum update shim-x64
Reboot to make sure machine still boots.

Message last modified on Aug 5, 2020 06:49PM

ev...@scailyte.com <ev...@scailyte.com> #30Aug 5, 2020 03:48PM

Were only instances with activated VTPM affected?

fa...@gmail.com <fa...@gmail.com> #31Aug 5, 2020 05:53PM

st...@google.com, thanks for you help on

comment #29

...solved!

st...@google.com <st...@google.com> #32Aug 5, 2020 06:50PM

Marked as fixed.

This issue is now locked. For a list of affected shim-64 versions, see comment #2. For a list of updated shim-64 versions, see comment #25.

For instructions on how to fix an instance that is currently unable to boot, see comment #29.