IssueTracker

I have included here what I have been able to copy.

I don't seem to have access. I requested access to one of them instead of requesting all of them to avoid spamming you with emails. Hopefully you can give me access to the folder they're in? Thanks!

I requested access to one of them instead of requesting all of them to avoid spamming you with emails.

I got access. Thanks! I looked over all the pictures twice but unfortunately I couldn't find the part of the log that I needed. :( I don't know if that's because your log somehow didn't have this information or if the relevant pictures got missed. Specifically we are looking for screens that talk about "auxfw" or "ps8805".

Another user did manage to send us logs with the relevant bits. If you still have it in the state where you can get logs, I'd be curious if yours matched. On their logs the relevant bits said:

Update auxfw 0
ps8805.0: SPI bus timeout after 1000ms
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
vb2api_auxfw_sync: vb2ex_auxfw_update() returned 0x10000001
vp2api_fail: Need recovery, reason: 0x30 / 0x1

I'd also be interested if you can describe the exact behavior you saw better. Specifically:

1. Many people describing this issue talk about the device "looping" or "blinking" that it's "applying a critical update". Did you see this too? ...but this I don't mean just a few loops/blinks, but (as I understand it) many users are saying that the device would just loop trying to apply the critical update for hours.

2. Assuming you saw the device looping/blinking, how much time (roughly) would you estimate was between each loop/blink? In other words, pick between:

• a) The "critical update" screen stayed up for 30 - 60 seconds, then the screen went black for a few seconds and then the "critical update" screen came back.

• b) The "critical update" screen stayed up for 1 - 2 seconds, then the screen went black for a few seconds and then the "critical update" screen came back.

• c) The "critical update" screen only stayed up for a very short amount of time (less than half a second) and you could barely read it before the screen went black for a few seconds and then the "critical update" screen came back.

• d) None of the above are right and something else happened.

3. In order to get the photos you attached, somehow the device must have stopped looping / blinking and you ended up at the "Something went wrong" screen, right? What caused the device to stop looping / blinking? Did it just stop on its own after a while (how long?), or did you try to run a recovery image (which one?), or maybe your device never looped/blinked and

I'd also like to note that we are still very interested in getting serial numbers associated with failing devices. We can use those to try to see if this issue correlates with any particular variants of the hardware that were shipped. That can help us to solve the problem more quickly. Just out of an abundance of caution, it's probably best not to post serial numbers directly in a public bug like this one. Probably the best way to provide serial numbers is to share them via Private Message to the official ChromeOS Reddit account. If, for some reason, you don't want to do that then yell and I can provide you my contact info.

Re comment#6: thank you for the pictures and the movie. That's incredibly useful and will help us make forward progress on this issue. Transcribing the relevant details:

ps8805.0: vendor 0x1da0 product 0x8805 device 0x0002 fw_rev 0x00
ps8805.1: vendor 0x1da0 product 0x8805 device 0x0002 fw_rev 0x21
vb2api_auxfw_sync: Updating auxfw
AUXFW is updating. Show firmware sync screen.
...
Update auxfw 0
ps8805.0: SPI bus timeout after 1000ms
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
vb2api_auxfw_sync: vb2ex_auxfw_update() returned 0x10000001
vb2api_fail: Need recovery, reason: 0x30 / 0x1

Unfortunately I didn’t find this post before doing the recovery so I couldn’t grab a video before.

No worries. It's useful to know that running the recovery image got you out of the looping phase to the "Something went wrong" phase. When you were in the looping phase, can you at least estimate if you were in a), b), c), or d) (from comment#4)? In other words, when it was looping did it sit there in the "critical update" screen for a long time (more than a few seconds), or was it a quick reboot?

Also: when you ran the recovery image, which recover image did you run? Did you use the one pointed to by the reddit post (which would have been R116) or did you just use a normal recovery image (which, I believe, would have been R117)?

If you are comfortable with it, I know folks working on the issue would really appreciate getting more serial numbers to help narrow down the issue. Please see comment#5.

Absolutely B. 1-2 seconds and then it would go black for a few seconds and then go back to the Critical Update for another 1-2, black again, etc.

This helps a ton, thanks! It's a bit past my normal working hours for today so I'll analyze more tomorrow, but knowing that the critical update stayed for 1-2 seconds instead of 30-60 seconds helps narrow things down a lot.

Good question. My first attempt at recovery was using whatever is automatically downloaded during the CRU download process. Probably R117. That's the one that stopped the loop and brought me to "Something went wrong". I later found an older version online, 113 I think, and tried that to no avail.

Interesting. I will ponder what this means tomorrow and see if I can figure anything out.

I'm happy to share the SN and will hop over to Reddit to see if I can DM it. Thanks!

Thank you! This is much appreciated.

Re comment#10: Got them. Transcribing yours:

sync_ec: select_rw=RW(active)
sync_ec: jumping to EC-RW
EC returned from reboot after 46943us
ps8805.0: vendor 0x1da0 product 0x8805 device 0x0002 fw_rev 0x00
ps8805.1: vendor 0x1da0 product 0x8805 device 0x0002 fw_rev 0x21
ps8805.1: PD_SUSPEND busy! Could be only power source.
vb2api_auxfw_sync: Updating auxfw
AUXFW is updating. Show firmware sync screen.
...
Update auxfw 0
ps8805.0: SPI bus timeout after 1000ms
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
ps8805.0: could not get chip info!
vb2api_auxfw_sync: vb2ex_auxfw_update() returned 0x10000001
vb2api_fail: Need recovery, reason: 0x30 / 0x1

The only differences in the logs here between yours and the others in this state just show that your device's battery was very low and it was running off the power cable you had plugged into the other port.

The video you sent matches the logs.

So it looks like all 3 of the firmware logs show devices in basically the same state when at the "Something went wrong" screen. We'll spend time today digging through for clues.

It would also be good to get answers to some of the questions from comment#4 to confirm that you were originally in the looping state and that you ended up at the "Something went wrong" state only after trying a recovery image. ...and to confirm that when you were at the looping state the "critical update" screen was up for ~1-2 seconds.

Also: if you haven't already sent your Serial Number to the official ChromeOS Reddit account and you're willing (see comment#5) that could still be useful to help us correlate components.

Sorry it's been quiet here for the past day or so. Despite that, we've been continuing to work on this issue internally, though there's nothing to report back yet. :(

A few overall updates:

• I believe we have enough of the firmware logs now, so people don't need to be bothered to scroll through the logs and take photos unless your logs are somehow unique from the ones already transcribed. If you have any doubt if yours is unique, I'm still happy to look at any logs posted to see if it gives us any extra clues.
• We still would appreciate getting serial numbers sent to the official ChromeOS Reddit account (see comment#5). These are used to try to correlate against manufacturer data to see if there's any way that the failing devices are related.
• It would still be useful to see a video of a device that's "looping" before the recovery image was run. We've got descriptions from several people (thanks!), but nothing beats a video. :-)

I would also note: if you aren't used to this bug tracker, you should make sure you click the "Star" in the upper left corner (you have to be signed into a Google account). That should make sure you get email updates about this bug.

Would it be useful to actually have one of the affected devices sent to you?

At the moment, we're still trying to figure out the best way forward. If that involves getting a device sent to us then we'll let you know.

Hello,

We are currently working on debugging this issue and we are asking users experiencing this issue to send us their Chromebooks, so we can further investigate this bug.

If you have an impacted Chromebook and would like to send it to our team free of charge, please send an email to: support-chromeos@google.com and we will provide you with further instructions.

Thank you, Doug - On behalf of ChromeOS

Also: in response to comment#17, we are still interested in videos of the "looping" mode since we never got one of those. We don't need any more videos of the "something went wrong" mode.

Manuel: I'm not directly involved in the process of shipping so I'm not sure whether being in the UK is a problem. The best thing to do is to email the support-chromeos@google.com address and they should be able to provide you with more details. Thanks!

Re comment#21: Sorry this issue has been silent for a bit. Rest assured we're still working on it! Let me give a quick update.

We've managed to get one device that was in the "Something Went Wrong" state and analyze it. As expected from all the logs provided by people (thanks!), the device was stuck updating the firmware of one of the components associated with the "Type C" port. We tried a number of ideas to try to make the update go through and talked to the vendor that provided the part but we had no luck there. The vendor has asked us to send the misbehaving part to them for failure analysis and we're hopeful that they will will provide a solution to allow us to recover these parts, but we're back in the waiting game here.

We do know for certain that replacing the daughterboard containing this chip fixes the problem. Replacing it on the unit we had in hand made the device boot normally again. For the curious about what the part looks like, I managed to find the part on Lenovo's website. The one we had to replace was on the left side so I believe it's "USB Board H 82QS Typc C_L" (part number 5C50S25268), there's no reason that we're aware of that the same issue couldn't affect the right side board which would be "USB Board H 82QS Typc C_R" (part number 5C50S25269).

While we're waiting for failure analysis (and hopefully a way to recover devices in this state), we continuing to look for other workarounds. If we come up with anything we'll post it here.

Hello Doug, really appreciate the prompt support you're giving us with this issue.

Just to confirm, as Chromebook Duet 5 and 3 are similar in many ways, can you confirm you've not received any reports of this failure happening on Duet 3?

Thank you!

Just to confirm, as Chromebook Duet 5 and 3 are similar in many ways, can you confirm you've not received any reports of this failure happening on Duet 3?

I'm not aware of any reports of this failure happening on Duet 3.