IssueTracker

I have included here what I have been able to copy.

I don't seem to have access. I requested access to one of them instead of requesting all of them to avoid spamming you with emails. Hopefully you can give me access to the folder they're in? Thanks!

I requested access to one of them instead of requesting all of them to avoid spamming you with emails.

I got access. Thanks! I looked over all the pictures twice but unfortunately I couldn't find the part of the log that I needed. :( I don't know if that's because your log somehow didn't have this information or if the relevant pictures got missed. Specifically we are looking for screens that talk about "auxfw" or "ps8805".

Another user did manage to send us logs with the relevant bits. If you still have it in the state where you can get logs, I'd be curious if yours matched. On their logs the relevant bits said:

Update auxfw 0
ps8805.0: SPI bus timeout after 1000ms
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
vb2api_auxfw_sync: vb2ex_auxfw_update() returned 0x10000001
vp2api_fail: Need recovery, reason: 0x30 / 0x1

I'd also be interested if you can describe the exact behavior you saw better. Specifically:

1. Many people describing this issue talk about the device "looping" or "blinking" that it's "applying a critical update". Did you see this too? ...but this I don't mean just a few loops/blinks, but (as I understand it) many users are saying that the device would just loop trying to apply the critical update for hours.

2. Assuming you saw the device looping/blinking, how much time (roughly) would you estimate was between each loop/blink? In other words, pick between:

• a) The "critical update" screen stayed up for 30 - 60 seconds, then the screen went black for a few seconds and then the "critical update" screen came back.

• b) The "critical update" screen stayed up for 1 - 2 seconds, then the screen went black for a few seconds and then the "critical update" screen came back.

• c) The "critical update" screen only stayed up for a very short amount of time (less than half a second) and you could barely read it before the screen went black for a few seconds and then the "critical update" screen came back.

• d) None of the above are right and something else happened.

3. In order to get the photos you attached, somehow the device must have stopped looping / blinking and you ended up at the "Something went wrong" screen, right? What caused the device to stop looping / blinking? Did it just stop on its own after a while (how long?), or did you try to run a recovery image (which one?), or maybe your device never looped/blinked and

I'd also like to note that we are still very interested in getting serial numbers associated with failing devices. We can use those to try to see if this issue correlates with any particular variants of the hardware that were shipped. That can help us to solve the problem more quickly. Just out of an abundance of caution, it's probably best not to post serial numbers directly in a public bug like this one. Probably the best way to provide serial numbers is to share them via Private Message to the official ChromeOS Reddit account. If, for some reason, you don't want to do that then yell and I can provide you my contact info.

Re comment#6: thank you for the pictures and the movie. That's incredibly useful and will help us make forward progress on this issue. Transcribing the relevant details:

ps8805.0: vendor 0x1da0 product 0x8805 device 0x0002 fw_rev 0x00
ps8805.1: vendor 0x1da0 product 0x8805 device 0x0002 fw_rev 0x21
vb2api_auxfw_sync: Updating auxfw
AUXFW is updating. Show firmware sync screen.
...
Update auxfw 0
ps8805.0: SPI bus timeout after 1000ms
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
vb2api_auxfw_sync: vb2ex_auxfw_update() returned 0x10000001
vb2api_fail: Need recovery, reason: 0x30 / 0x1

Unfortunately I didn’t find this post before doing the recovery so I couldn’t grab a video before.

No worries. It's useful to know that running the recovery image got you out of the looping phase to the "Something went wrong" phase. When you were in the looping phase, can you at least estimate if you were in a), b), c), or d) (from comment#4)? In other words, when it was looping did it sit there in the "critical update" screen for a long time (more than a few seconds), or was it a quick reboot?

Also: when you ran the recovery image, which recover image did you run? Did you use the one pointed to by the reddit post (which would have been R116) or did you just use a normal recovery image (which, I believe, would have been R117)?

If you are comfortable with it, I know folks working on the issue would really appreciate getting more serial numbers to help narrow down the issue. Please see comment#5.

Absolutely B. 1-2 seconds and then it would go black for a few seconds and then go back to the Critical Update for another 1-2, black again, etc.

This helps a ton, thanks! It's a bit past my normal working hours for today so I'll analyze more tomorrow, but knowing that the critical update stayed for 1-2 seconds instead of 30-60 seconds helps narrow things down a lot.

Good question. My first attempt at recovery was using whatever is automatically downloaded during the CRU download process. Probably R117. That's the one that stopped the loop and brought me to "Something went wrong". I later found an older version online, 113 I think, and tried that to no avail.

Interesting. I will ponder what this means tomorrow and see if I can figure anything out.

I'm happy to share the SN and will hop over to Reddit to see if I can DM it. Thanks!

Thank you! This is much appreciated.

Re comment#10: Got them. Transcribing yours:

sync_ec: select_rw=RW(active)
sync_ec: jumping to EC-RW
EC returned from reboot after 46943us
ps8805.0: vendor 0x1da0 product 0x8805 device 0x0002 fw_rev 0x00
ps8805.1: vendor 0x1da0 product 0x8805 device 0x0002 fw_rev 0x21
ps8805.1: PD_SUSPEND busy! Could be only power source.
vb2api_auxfw_sync: Updating auxfw
AUXFW is updating. Show firmware sync screen.
...
Update auxfw 0
ps8805.0: SPI bus timeout after 1000ms
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
vb2api_auxfw_sync: vb2ex_auxfw_update() returned 0x10000001
vb2api_fail: Need recovery, reason: 0x30 / 0x1

The only differences in the logs here between yours and the others in this state just show that your device's battery was very low and it was running off the power cable you had plugged into the other port.

The video you sent matches the logs.

So it looks like all 3 of the firmware logs show devices in basically the same state when at the "Something went wrong" screen. We'll spend time today digging through for clues.

It would also be good to get answers to some of the questions from comment#4 to confirm that you were originally in the looping state and that you ended up at the "Something went wrong" state only after trying a recovery image. ...and to confirm that when you were at the looping state the "critical update" screen was up for ~1-2 seconds.

Also: if you haven't already sent your Serial Number to the official ChromeOS Reddit account and you're willing (see comment#5) that could still be useful to help us correlate components.

Sorry it's been quiet here for the past day or so. Despite that, we've been continuing to work on this issue internally, though there's nothing to report back yet. :(

A few overall updates:

• I believe we have enough of the firmware logs now, so people don't need to be bothered to scroll through the logs and take photos unless your logs are somehow unique from the ones already transcribed. If you have any doubt if yours is unique, I'm still happy to look at any logs posted to see if it gives us any extra clues.
• We still would appreciate getting serial numbers sent to the official ChromeOS Reddit account (see comment#5). These are used to try to correlate against manufacturer data to see if there's any way that the failing devices are related.
• It would still be useful to see a video of a device that's "looping" before the recovery image was run. We've got descriptions from several people (thanks!), but nothing beats a video. :-)

I would also note: if you aren't used to this bug tracker, you should make sure you click the "Star" in the upper left corner (you have to be signed into a Google account). That should make sure you get email updates about this bug.

Would it be useful to actually have one of the affected devices sent to you?

At the moment, we're still trying to figure out the best way forward. If that involves getting a device sent to us then we'll let you know.

Hello,

We are currently working on debugging this issue and we are asking users experiencing this issue to send us their Chromebooks, so we can further investigate this bug.

If you have an impacted Chromebook and would like to send it to our team free of charge, please send an email to: support-chromeos@google.com and we will provide you with further instructions.

Thank you, Doug - On behalf of ChromeOS

Also: in response to comment#17, we are still interested in videos of the "looping" mode since we never got one of those. We don't need any more videos of the "something went wrong" mode.

Manuel: I'm not directly involved in the process of shipping so I'm not sure whether being in the UK is a problem. The best thing to do is to email the support-chromeos@google.com address and they should be able to provide you with more details. Thanks!

Re comment#21: Sorry this issue has been silent for a bit. Rest assured we're still working on it! Let me give a quick update.

We've managed to get one device that was in the "Something Went Wrong" state and analyze it. As expected from all the logs provided by people (thanks!), the device was stuck updating the firmware of one of the components associated with the "Type C" port. We tried a number of ideas to try to make the update go through and talked to the vendor that provided the part but we had no luck there. The vendor has asked us to send the misbehaving part to them for failure analysis and we're hopeful that they will will provide a solution to allow us to recover these parts, but we're back in the waiting game here.

We do know for certain that replacing the daughterboard containing this chip fixes the problem. Replacing it on the unit we had in hand made the device boot normally again. For the curious about what the part looks like, I managed to find the part on Lenovo's website. The one we had to replace was on the left side so I believe it's "USB Board H 82QS Typc C_L" (part number 5C50S25268), there's no reason that we're aware of that the same issue couldn't affect the right side board which would be "USB Board H 82QS Typc C_R" (part number 5C50S25269).

While we're waiting for failure analysis (and hopefully a way to recover devices in this state), we continuing to look for other workarounds. If we come up with anything we'll post it here.

Hello Doug, really appreciate the prompt support you're giving us with this issue.

Just to confirm, as Chromebook Duet 5 and 3 are similar in many ways, can you confirm you've not received any reports of this failure happening on Duet 3?

Thank you!

Just to confirm, as Chromebook Duet 5 and 3 are similar in many ways, can you confirm you've not received any reports of this failure happening on Duet 3?

I'm not aware of any reports of this failure happening on Duet 3.

We apologize for the inconvenience this has caused, we are working with our partners to resolve this issue as quickly as possible. A small subset of users are experiencing this bug, which is why we have decided to pause auto-updates until we can roll out a fix for this issue. Please see below for further details please see below.

We would like to share further updates about this issue. Currently we're waiting for our vendor to perform failure analysis on the affected component.

As people have noticed, auto-updates for the sibling devices, Duet 5 (homestar) and Duet 3 (wormdingler), were paused in association with this issue. Specifically, Duet 5 was paused at R116 and Duet 3 was paused at R117. This is because R117 contained the firmware update associated with this issue for the Duet 5 and R118 contained the firmware update associated with this issue for Duet 3.

For Duet 5, the majority of users successfully updated to R117 before the issue was identified. Anyone who made it to R117 on Duet 5 stayed there for a while as we worked on the issue but those that made it to R117 have now been cleared to receive R118. R118 has the same firmware for Duet 5 as R117 so it's safe for Duet 5 users on R117 to update to R118. Those on R116 are still held on R116.

For Duet 3, R118 was never pushed to the stable channel and all Duet 3 devices are being held to R117. It can be noted that there are no actual reports of Duet 3 devices being affected by this issue, but since the machine is very similar to the Duet 5 hardware-wise we've been holding the updates in an abundance of caution.

Note that only a very small number of users who get the firmware update are affected by this issue, which is the whole reason that R117 was pushed to so many people on Duet 5 before the problem was identified and also why debugging has been so difficult. Despite only affecting a very small number of users, we have still been holding updates for this because we don't want any more devices to run into it. While we realize that nobody wants their update held, we believe this is the right choice for users.

I recently purchased a Duet 5 myself and will share some observations that might be relevant:

1. It was manufactured on August 5th of this year, shipped with an outdated version of ChromeOS, which I updated to 105.0.5195.134, before updating it to 116.0.5845.210
2. I then powered it off with nothing connected to either of the USB-C ports, powered it back on, and switched to the Beta channel
3. After updating to 120.0.6099.25, it showed the "Your system is applying a critical update." screen, then rebooted some time later, before displaying the ChromeOS logo, and reaching the login screen
4. After the update, I have observed that sometimes, if I attach a device to either of the USB-C ports, said device isn't detected, and the only way to fix it is rebooting. (I did not test this before the update however)

For those Duet5's currently on 117 when do you anticipate the 118 push to be received?

I would expect that if you check for updates that 118 should be available for Duet5's currently on 117.

Just a short update for this following this bug so people know what to expect.

We are currently working on landing a mitigation for this issue. The mitigation will include a firmware that skips trying to update the Type C component that was causing the issue. Once the mitigation lands:

• We will be able to unblock updates for both affected boards (Duet 3 and Duet 5).
• Anyone who has a device that is stuck in the "Something went wrong" screen will be able to download/boot a ChromeOS recovery image which will make their device bootable again. NOTE: if your device was at the "Something went wrong" screen then one of the two Type C ports may only be partially functional until the final fix is ready.

We've also received a positive result from the failure analysis performed by our vendor. The vendor has identified the problem and has provided us with a fix. We have validated that the fix is able to fully recover the failing device that we had and are starting to kick off the processes needed to get this final fix out. Once the final fix is out then everyone's Type C component should get updated to the newest version and anyone whose device was at the "Something went wrong" screen because of this issue should again have two functioning Type C ports.

I cannot promise any particular dates here, but I wanted to at least let folks know that an end is in sight. I'll post again when there is more news to share.

comment#36 -

Your devices sound like they are experiencing a different failure. The mitigation is currently focused on 'homestar' platform. There are multiple Lenovo 100e Gen 2 variants, but none share the same platform.

Can you provide logs? If it is different another bug would be helpful so that the two failures are not conflated.

If someone with a Duet 5 (or Duet 3 for that matter since they're the same board) is on the Beta Channel that's now on ChromeOS 120 at the time of this writing, is it safe to assume the device isn't affected?

Yes.

Is there an ETA, given the last update was 10 days ago?

We're almost there. Hang tight!

thx for the update. now on version 119. thank you for fixing this issue.

Yup, you beat me to it. I'm told that R119 should be starting to roll out now. Recovery images are still not posted for those who are still stuck at the "Something went wrong" screen and need to recover. I'll post again when that's available.

For anyone that's stuck at the bootloop or the "Something went wrong" screen, we have an option available for you. If you use the Chromebook Recovery Utility we've temporarily added the ability to download a recovery image for the "Beta" channel for "homestar" devices. In order to get this you need to go to the "Show Advanced Settings" in the tool and choose the "Beta" channel.

This will get you on the R120 "beta" channel (15662.35.0) and get you the new firmware containing the mitigation. This should get people booting again, though you'll be on beta channel.

If you don't want to be on beta channel then you'll need to hang tight a little longer until the R119 recovery images are published. I'll post again here when that happens.

Thanks! Will you leave the option available until after the holidays?

We won't take the beta channel option down until there is a stable channel option available. Unless something goes sideways (always possible) that should be very soon.

I won't have access to a windows computer to try it out before then.

The recovery utility should be able to run from other Chromebooks as well if that helps. I think (but haven't checked) that it also runs from Macs.

We won't take the beta channel option down until there is a stable channel option available. Unless something goes sideways (always possible) that should be very soon.

Good news! Stable channel recovery images are now available, so you should be able to use the Chromebook Recovery Utility to make a normal recovery image and it should get you on R119 (15633.65.0). As a reminder: the only people who need to do this are those who were stuck at the "Something went wrong screen" or were bootlooping applying the critical update. Everyone else should get R119 as part of the normal autoupdate process.

I will continue to leave this bug open to let people know when the final fix comes out. If one of your two Type C ports lost functionality as part of this issue it will be recovered by the final update. It may be a while before this final fix gets released as the validation process will be slowed down by the holiday season.

Thank you to everyone for their patience as we resolve this issue.

my duet is still on the applying critical update loop

FWIW, we never did get a video of this stage and still have never seen it ourselves. If you are willing, we certainly wouldn't object to a video of the looping state.

do i need to put it in some sort of recovery mode before i plug in the hard drive?

Yes. There should be instructions for entering recovery mode on our support site. You'd want to use the instructions for a "Chromebook tablet", which are:

Press and hold the Volume Up, Volume Down, and Power buttons for at least 10 seconds, then release them

As a heads-up to anyone watching this bug, I'm going to be on vacation for the next few weeks. Qualification of the final fix continues, though (as I said earlier) it can take a while. I'll post again here when there is more news on that front.

I have a duet 5 stuck in the boot looping and non of the key combinations will work to get into any utility. Just keeps looping. Any ideas?

Just to confirm: you're following the instructions for Chromebook tablet from our support site, right? Devices that have a detachable keyboard can't use the laptop-style instructions even if the keyboard is attached.

On a separate note: the fixed firmware has landed on R122 and R122 is currently being served on dev channel. If your type C port was affected by this issue, it's possible to get the updated firmware which should recover your type C port. If you are interested in doing this, you should be able to do:

1. Switch to dev channel in the UI.
2. Wait for the dev channel image to be downloaded.
3. Reboot and you should see the "critical update" screen run
4. Once everything is good and you've finished updating, you could use a recovery image to get back to stable channel and the type C port will remain recovered (like all recovery images, this will erase any local data).

I mention the steps above only if someone is very eager to get their hands on the update. I would expect that the majority of the people can wait until the update makes it out to the stable channel.

I'll plan to post again and close this bug when R122 has made it to stable.

Can anyone offer tips as to why I get a message saying 'Download interrupted. Please check your network connection and try again'?

This is probably not the right place for general Chromebook help. I believe the Chromebook Help Community is probably more appropriate? When I search for "Download interrupted. Please check your network connection and try again" on Google I find at least a few other people talking on that site and people there seem to indicate that a slow flash drive may be the problem. I guess my suggestion would be to try to see if you can find a friend to help you create the recovery image. Maybe you can borrow yet another different flash drive, or perhaps use a different computer to run the recovery utility to see if that helps? If that doesn't work, I'd suggest you ask on the help community site.

Is it possible that Google will roll out a final fix that will automatically fix this issue? Or we will still need to go through the recovery image process?

Unfortunately if you're device is currently not booting then there's no way for us to roll out a fix to you. You'll have to go through the recovery image process.

I did try using both a PNY and SanDisk USB but I will try one more brand

It might also be interesting to try using a different computer to run the recovery utility. It's always possible that there's some bad interaction with the computer that's running the recovery utility.

I'll plan to post again and close this bug when R122 has made it to stable.

It appears that R122 is released to stable. Folks should be able to check for updates and get the new version which is expected to fix Type C ports that were affected by this issue. Just to be on the safe side, it wouldn't hurt to make sure your battery is well charged before taking the update. ;-)

Closing this bug now. Thanks for holding tight and apologies again to anyone who was impacted by this issue.

since it has been released my duet 5 stopped downloading updates. "could't download the update, please try again later" I am on version 121.0.6167.188.

Sorry you're having trouble! This is pretty unlikely to be related to the problem that this bug is about. Probably a better place to go for help would be the Chromebook Help Community. I would imagine that folks there would be able to help assist with troubleshooting.

I am now having this issue as well. My duet 5 said the same “your system is applying a critical update, do not turn off”, kept flashing black, and now will not turn on or boot into recovery mode, despite being connected to a power source. I used it recently, so it should have been on a recent version of Chrome OS. Would anyone be able to help me? Thank you.

Sorry you're having trouble.

If it can't boot into recovery mode then there's not much I can suggest other than troubleshooting steps to get it to turn on again:

• Try several different power cords. It could be worth it to try both the power cable that came with the laptop as well as more basic power supplies.
• Try both of the Type-C ports.
• Try connecting power cords directly (not through a dock).

Thanks for the picture and video. That helps us understand what's going on.

The bad news, though, is that I'm not sure that I can come up with anything that'll fix it. My best guess is that somehow:

• On each bootup the device tries to update the a Type C port and it thinks it succeeded (so you don't get the "Something went wrong" screen). Then it reboots and somehow the update didn't take, so it tries again. It keeps doing this.
• If the Type C port is in a state with an incomplete update then it probably can't recognize USB disks, which is probably why your recovery disks aren't working.
• If USB disks aren't working on both ports then somehow (?) both of the ports must have an incomplete update.

I'm guessing that somehow your system must have decided to update to an earlier version of the firmware where this was a problem. We believe that the problem is fixed with the newest version (and this is the only report I've heard since we released the fix). I don't know of any easy way to confirm this from the state your system is in, though. :(

If your device is still under warranty I'd suggest that you exercise that option as it would be the easiest way to get the problem resolved.

If you're not under warranty then your options are what I could only classify as bad (though maybe better than throwing the device away). Please don't take this as official advice and these solutions are pretty technical / difficult. Options I can think of:

1. If you can find someone comfortable with opening up your device, if you unplug the two "Type C" daugherboards then the device should boot (assuming the batteries have enough charge). Ideally it could get autoupdate and get another firmware update and the new firmware update might be able to handle you plugging the Type C ports back in. This all assumes that, for some reason, the firmware you have right now isn't the newest. If you find someone comfortable with this (like a repair shop) it does look possible to order replacement daughtercards (see comment#22).
2. If someone was a software developer and somehow got a CCD cable (which is hard to do) there might be steps you could do to tell the bootloader not to update the Type C port. This would likely be quite a long journey, though.

Sorry for not having better news. :(

I disconnected the daughter boards and was able to successfully boot and update to the latest update. It looks like I was on 128.0.6613.161

That's mysterious since 128.0.6613.161 should have had the newest version of the bootloader. ...and the fact that you have the newest bootloader is extra confirmed by the fact that on the next reboot it didn't loop and was able to update OK. I suspect we'll never be able to figure out what caused the issue you were seeing, but I'm glad at least you're booting again!

However, it looks like my right USB port is not working/receiving power, only the left one. Is there any way I can fix the right daughter board?

I guess I'll start with the question of whether you've tried disconnecting and reconnecting the board again. Since you said you disconnected both boards and reconnected them it's possible that you didn't quite get the cable pushed all the way in or aligned properly when you reconnected one side.

If that doesn't help, you might be stuck. If you want, you could make sure that the board is connected, boot up, and file a feedback report (including logs) with some obvious text in it like "#issue306756143" and then let me know. I should be able to locate that and I can see if the logs show anything helpful, though I'm not that confident. If the board simply isn't responding then the logs won't help much.

What a nice love story :3

#73 Thanks to your troubleshooting efforts I'm confident that I'll buy Chromebooks for myself, my family and friends going forward.

This issue should be kept as a reference within Google on how to support customers in a professional manner that will keep them loyal to you.

You deserve a bonus (and should lecture the Android team about this).

0837124746. password. Face BOOK

Thailand 🇹🇭