mac reboots when emulator is closed. I have latest emulator installed [120951634]

Fixed

Bug

NeedsInfo

Status Update

No update yet.

Description

sa...@gmail.com

created issue #1

Dec 13, 2018 06:10PM

Build: 3.2.1, AI-181.5540.7.32.5056338, 201810090144,

AI-181.5540.7.32.5056338, JRE 1.8.0_152-release-1136-b06x64 JetBrains s.r.o, OS Mac OS X(x86_64) v10.14.1 unknown, screens 1440x900

Android Gradle Plugin: 3.2.1
Gradle: 4.6
NDK: from local.properties: 18.1.5063045; latest from SDK: 18.1.5063045;
LLDB: pinned revision 3.1 not found; latest from SDK: (package not found);
CMake: from local.properties: (not specified); latest from SDK: (not found); from PATH: (not found);

IMPORTANT: Please read

https://developer.android.com/studio/report-bugs.html carefully and supply all required information.

Comments

uc...@google.com <uc...@google.com> #2Dec 13, 2018 11:04PM

Assigned to lf...@google.com.

Adding 2 more thread dumps from another freeze. Again after switching branches and updating from git.

de...@googlemail.com <de...@googlemail.com> #3Dec 14, 2018 12:02AM

Thanks for the report. I believe that this is fixed in our development branch, and also in the branch that will become Ladybug Feature Drop Beta. If you could verify that this doesn't happen in a nightly build (from https://developer.android.com/studio/nightly) that would be helpful -- and of course if you can trigger this in a recent nightly build, that's would be very important information.

lf...@google.com <lf...@google.com> #4Dec 14, 2018 12:04AM

Sorry, I have not tried it on the nightly build, but since then I updated to Meerkat and I have not seen the freeze afterwards.

sa...@gmail.com <sa...@gmail.com> #5Dec 14, 2018 03:36AM

That's good to hear. I think that the fix (which for reference was https://github.com/JetBrains/intellij-community/commit/fe621544dd61a9af34d97958d3d823a7c3454a87) is effective for both beta and canary branches.

For the record I am also arranging for https://github.com/JetBrains/intellij-community/commit/a444c8a500883669fc05c7177f93b5b794fdd13b (same fix, different implementation class) to be included; it's probably inactive code in our configurations, but it might be possible to trigger nevertheless.

Thanks for the confirmation!

lf...@google.com <lf...@google.com> #6Dec 14, 2018 03:59AM

Thank you for your patience while our engineering team worked to resolve this issue. A fix for this issue is now available in:

Android Studio Ladybug Feature Drop | 2024.2.2 RC 1
Android Gradle Plugin 8.8.0-rc01

We encourage you to try the latest update.

If you notice further issues or have questions, please file a new bug report.

Thank you for taking the time to submit feedback — we really appreciate it!

ak...@gmail.com <ak...@gmail.com> #7Dec 14, 2018 01:23PM

Ucfucuccj

lf...@google.com <lf...@google.com> #8Dec 14, 2018 01:53PM

Hyxucyccchccjch

Message last modified on Dec 14, 2018 01:57PM

ak...@gmail.com <ak...@gmail.com> #9Dec 14, 2018 02:15PM

Ycucuccjcn

Message last modified on Dec 14, 2018 02:18PM

ak...@gmail.com <ak...@gmail.com> #10Dec 14, 2018 02:28PM

Hola mundo

lf...@google.com <lf...@google.com> #11Dec 14, 2018 02:41PM

#10 Great! We currently cannot reproduce this problem on our machines, but in order to rule things out, would you be willing to enable the feature again (set to " = on"), but use -gpu swiftshader_indirect?

lf...@google.com <lf...@google.com> #12Dec 14, 2018 02:43PM

This is because it could be an interaction between the feature and graphics acceleration:

com.apple.iokit.IOAcceleratorFamily2(400.27)[5BC3E111-E908-36E8-9AFB-A39BD0D561D5]@0xffffff7f902aa000->0xffffff7f9034efff
dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[66A1BDE1-514B-3B0D-87DD-8E2B9F110A8C]@0xffffff7f8ffb9000
dependency: com.apple.iokit.IOSurface(255.1)[B153C6D6-2542-3D6F-A21F-B56D9E1349DF]@0xffffff7f90227000
dependency: com.apple.iokit.IOPCIFamily(2.9)[7DE930EC-AB73-3F5A-9D38-94D5EE90D197]@0xffffff7f8fa95000
dependency: com.apple.iokit.IOGraphicsFamily(530.12)[9D1E5FA7-267E-397E-9C91-1751221A05CF]@0xffffff7f9024b000
dependency: com.apple.iokit.IOReportFamily(47)[6A9FF3E1-E644-3F33-8A44-8B51C44AA67C]@0xffffff7f902a3000
com.apple.driver.AppleIntelHD5000Graphics(12.0.2)[D8908CD8-D54B-3FD5-B1CF-76BB24CBF2EA]@0xffffff7f917fa000->0xffffff7f91874fff
dependency: com.apple.iokit.IOSurface(255.1)[B153C6D6-2542-3D6F-A21F-B56D9E1349DF]@0xffffff7f90227000
dependency: com.apple.iokit.IOPCIFamily(2.9)[7DE930EC-AB73-3F5A-9D38-94D5EE90D197]@0xffffff7f8fa95000
dependency: com.apple.iokit.IOGraphicsFamily(530.12)[9D1E5FA7-267E-397E-9C91-1751221A05CF]@0xffffff7f9024b000
dependency: com.apple.iokit.IOAcceleratorFamily2(400.27)[5BC3E111-E908-36E8-9AFB-A39BD0D561D5]@0xffffff7f902aa000

ak...@gmail.com <ak...@gmail.com> #13Dec 14, 2018 03:14PM

#11 It shuts down fine, just to confirm, I followed these steps:

echo "QuickbootFileBacked = on" >> ~/.android/advancedFeatures.ini
~/Library/Android/sdk/emulator/emulator -avd P1 -gpu swiftshader_indirect

lf...@google.com <lf...@google.com> #14Dec 14, 2018 03:27PM

#13: Thanks a lot, that helps narrow it down!

ak...@gmail.com <ak...@gmail.com> #15Dec 14, 2018 03:28PM

Glad to be able to help !

lf...@google.com <lf...@google.com> #16Dec 14, 2018 05:41PM

Can you try this emulator:

https://drive.google.com/open?id=1nT99ePCFNHC-EdG6qtlP9zzAuOyqKE6h

with host gpu and the feature enabled? It has a speculative fix that tries to unmap the ram file cleanly before emulator exit handlers run.

lf...@google.com <lf...@google.com> #17Dec 14, 2018 06:14PM

(just updated it again with a new version that should be faster)

ap...@google.com <ap...@google.com> #18Dec 14, 2018 06:54PM

Project: platform/external/qemu
Branch: emu-master-dev

commit 6241395508b6c846566fd423d83cd5ec607590ab
Author: Lingfeng Yang <lfy@google.com>
Date: Fri Dec 14 09:07:53 2018

[snapshot] Eagerly tear down RAM mappings on exit

bug: 120951634

Some users are experiencing kernel panic on snapshot save with
file-backed RAM when -gpu host is also enabled; the graphics driver
seems to cause a page fault in the kernel when saving on exit.

Speculative fix: Eagerly tear down RAM mappings.

+ a subtlety: Don't run ram block removal notifiers when exiting;
it's possible they were registered to a virtual device that has also
been torn down, possibly segfaulting in user space.

Change-Id: Iee1567dac2c997f1c89a573cb2e5a7369a1399ad

M exec.c
M hw/i386/pc.c
M hw/i386/pc_piix.c
M include/exec/memory.h
M include/exec/ram_addr.h
M include/hw/boards.h
M include/hw/i386/pc.h
M memory.c
M vl.c

https://android-review.googlesource.com/850227

https://goto.google.com/android-sha1/6241395508b6c846566fd423d83cd5ec607590ab

ap...@google.com <ap...@google.com> #19Dec 14, 2018 06:55PM

Project: platform/external/qemu
Branch: emu-3.0-release

commit f74d5e2da6b477c969ed27fa984497a87008ccd2
Author: Lingfeng Yang <lfy@google.com>
Date: Fri Dec 14 09:07:53 2018

[snapshot] Eagerly tear down RAM mappings on exit

bug: 120951634

Some users are experiencing kernel panic on snapshot save with
file-backed RAM when -gpu host is also enabled; the graphics driver
seems to cause a page fault in the kernel when saving on exit.

Speculative fix: Eagerly tear down RAM mappings.

+ a subtlety: Don't run ram block removal notifiers when exiting;
it's possible they were registered to a virtual device that has also
been torn down, possibly segfaulting in user space.

Change-Id: Iee1567dac2c997f1c89a573cb2e5a7369a1399ad
(cherry picked from commit 6241395508b6c846566fd423d83cd5ec607590ab)

M exec.c
M hw/i386/pc.c
M hw/i386/pc_piix.c
M include/exec/memory.h
M include/exec/ram_addr.h
M include/hw/boards.h
M include/hw/i386/pc.h
M memory.c
M vl.c

https://android-review.googlesource.com/850929

https://goto.google.com/android-sha1/f74d5e2da6b477c969ed27fa984497a87008ccd2

lf...@google.com <lf...@google.com> Dec 14, 2018 10:19PM

Accepted by lf...@google.com.

lf...@google.com <lf...@google.com> #20Dec 14, 2018 11:37PM

For anyone else having this problem, can you try latest canary emulator 28.0.21 with the feature enabled?

ur...@gmail.com <ur...@gmail.com> #21Dec 15, 2018 07:34AM

Happen on my mac too. I'm on 28.0.20 *Stable* emulator and macOs Mojave 10.14.2

lf...@google.com <lf...@google.com> #22Dec 17, 2018 03:30PM

#21: Can you try emulator 28.0.21 in the canary channel?

ma...@gmail.com <ma...@gmail.com> #23Dec 18, 2018 11:42PM

It fixes the problem for when started with echo "QuickbootFileBacked = on" >> ~/.android/advancedFeatures.ini and
~/Library/Android/sdk/emulator/emulator -avd Nexus_5X_API_28 -gpu swiftshader_indirect

The funny thing is, that after deleting the advancedFeatures.ini and starting it without the gpu parameter, it keeps working correctly (it printed WARNING: change of renderer detected. after I removed the gpu parameter, so maybe it still uses the previously provided one)

I had the same problem with the emulator in version 27.x, so I'm very happy to see it working now.

https://issuetracker.google.com/issues/118849064 - probably you can mark it as a duplicate

sa...@gmail.com <sa...@gmail.com> #24Dec 19, 2018 03:07AM

Hi I am still having that issue, I am a beginner programmer, Can you please
tell me how to start emulator with these commands.

On Wed, Dec 19, 2018 at 5:13 AM <buganizer-system@google.com> wrote:

- Show quoted text -

--

*Sachin Malik*

*Android Developer - Caringforyou IT solutions*

Call: +91 <

http://nitinsinhame.net/>8802729349 | Mail:
sachinmalik413@gmail.com

Please consider the environment before printing this email

lf...@google.com <lf...@google.com> #25Dec 19, 2018 03:17AM

#23: It will use host gpu if the gpu parameter is not specified. Can you also try cold booting it and then saving state there? Also, was this on 28.0.21 canary?

#24: Can you try the latest emulator canary, 28.0.21? You can update to it by going to Studio preference > Updates section, and selecting Canary, then going back to the SDK Manager > Tools tab and updating Emulator to 28.0.21.

ma...@gmail.com <ma...@gmail.com> #26Dec 19, 2018 04:17AM

After I downloaded 28.0.21 I just tried it straight away without and parameters and it crashed the system. Then I tried the one with gpu, it was fine. It's been fine after that so far, although I don't specify the gpu any more. Cold start, wiping data, restarting the computer, different image, upgrading to macos 10.14.2 still not crashing. Which is good, just a bit strange. The version of the emulator is 28.0.21-5183090 (HVF 10.14.0)

lf...@google.com <lf...@google.com> #27Dec 19, 2018 04:20AM

#26: Ok, yeah that is a bit strange then, though it seems like at least it's not happening every time for you? It's interesting it also happened in 27.x; definitely feels more like specifically a GPU driver issue now, perhaps the delay on exit matters---having the faster snapshot save seemed to have exposed an issue at that level.

ma...@gmail.com <ma...@gmail.com> #28Dec 19, 2018 04:27AM

Before it was happening almost always, I think the only one time it didn't happen was when the system had plenty of free RAM (there was only AS running and the emulator). If I see the crash again I'll let you know in this thread. (And the mac I'm using is the 2015 macbook pro without a discreet graphics card, it has only Intel Iris Pro).

lf...@google.com <lf...@google.com> #29Dec 19, 2018 04:29AM

#28: Ok, thanks. It's weird since I also have a 2015 macbook pro, but ran a script to start/exit with quickboot save 1000x inside google maps and didn't happen once. Does this happen with some apps more than others?

ma...@gmail.com <ma...@gmail.com> #30Dec 19, 2018 04:36AM

Haven't seen any pattern with the app.

na...@gmail.com <na...@gmail.com> #31Dec 19, 2018 02:34PM

The crash happened to me on my new Mac Mini with Mojave 14.2 and Emulator 28.0.20. I just switched to Mac recently, and imported my AS settings from Windows (probably not related but just in case). I reproduced it twice yesterday, then created a new emulator with the same settings today and it didn't crash on exit.

Build: 3.3 RC 2, AI-182.5107.16.33.5160847, 201812040121,

AI-182.5107.16.33.5160847, JRE 1.8.0_152-release-1248-b01x64 JetBrains s.r.o, OS Mac OS X(x86_64) v10.14.2 unknown, screens 1920x1080; Retina

Android Gradle Plugin: 3.3.0-rc02
Gradle: 4.10.1
NDK: from local.properties: (not specified); latest from SDK: (not found);
LLDB: pinned revision 3.1 not found; latest from SDK: (package not found);
CMake: from local.properties: (not specified); latest from SDK: (not found); from PATH: (not found);

Message last modified on Dec 19, 2018 02:34PM

yo...@gmail.com <yo...@gmail.com> #32Dec 19, 2018 02:39PM

Same issue on Macbook Pro 13" 2017 with emulator 28.0.20. Previous versions, no problem.

lf...@google.com <lf...@google.com> #33Dec 19, 2018 03:43PM

#31, #32: Please try the emulator canary 28.0.21 and report if it makes a difference.

na...@gmail.com <na...@gmail.com> #34Dec 19, 2018 04:02PM

My machine just crashed twice with 28.0.21. I'm not seeing much of a pattern. It's both worked and not worked with .20 and .21

I would keep testing, but I'm not very keen on kernel panics. I would be willing to do it a couple of time if there are some sort of error logs we could send.

lf...@google.com <lf...@google.com> #35Dec 19, 2018 04:46PM

#34 Are you getting any kind of kernel panic log as in

comment #7

lf...@google.com <lf...@google.com> #36Dec 19, 2018 08:56PM

Everyone who can reproduce this problem: Try out the linked emulator by running from the command line and piping all output to a file:

https://drive.google.com/open?id=1pHXfTX-1m16gO9MVPTAWWHJ4suLlFzsj

objs-mac-debug-log/emulator -avd <avdname> &> log.txt

and post the log.

lf...@google.com <lf...@google.com> #37Dec 20, 2018 03:22AM

Another build with more lightweight debug logs:

https://drive.google.com/open?id=1zaIFuQ3iNwQlaBM4thk_JathzvZe8WZx

na...@gmail.com <na...@gmail.com> #38Dec 20, 2018 01:53PM

Here are logs from my previous crashes. They are not the same format as the logs in #7. I grabbed them from /Library/Logs/DiagnosticReports/ProxiedDevice-Bridge.

I'll try the emulator from #37 today.

Archive.zip

899 KB

Download

na...@gmail.com <na...@gmail.com> #39Dec 20, 2018 02:06PM

Crashed again with the build with lightweight logs. They don't seem to have much, but here you go. I'll try the emulator from #36 now.

log.txt

837 B

View

Download

na...@gmail.com <na...@gmail.com> #40Dec 20, 2018 02:13PM

Here's a log from when it crashed with the emulator from #36

log.txt

837 B

View

Download

lf...@google.com <lf...@google.com> #41Dec 20, 2018 06:12PM

Thanks very much for the logs! Although they are different, it's still essentially in the Intel driver.

We might have a fix that tries to avoid that code path in Apple's Intel GPU driver.

Try the emulator at this link:

https://drive.google.com/open?id=1z46V3UdtXEZgXBEGF6Lf2OLb_3n7jUdY

lf...@google.com <lf...@google.com> #42Dec 20, 2018 06:14PM

#39, #40: I have since found a consistent repro: V1 Pro graphics benchmark; start it up, and while loading, quickboot save the emulator. Thus, I also found that piping it to the log may not be of much help as on a kernel panic, the log is still buffered and it gets cut off. One may have to film the monitor for the correct error log.

lf...@google.com <lf...@google.com> #43Dec 20, 2018 06:54PM

And this one (should also fix it, but be a smaller download):

https://drive.google.com/open?id=185KRHEaYOUyLuNEKqQu96aRCu_1-s_x3

ap...@google.com <ap...@google.com> #44Dec 20, 2018 07:33PM

Project: platform/external/qemu
Branch: emu-master-dev

commit 6286c8ca1fd2fe7d264500522c1b18a63effba2e
Author: Lingfeng Yang <lfy@google.com>
Date: Thu Dec 20 08:46:02 2018

[snapshot][gl] Always read some formats with glReadPixels

bug: 120951634

This CL makes texture images with the following setup:

- GL_TEXTURE_2D
- level 0
- GL_RGB(A)
- GL_UNSIGNED_BYTE

to be read with glReadPixels.

This is meant to work around a macOS kernel panic.

Test: V1 Benchmark Pro texture save does not cause kernel panic on Mac
Test: Snapshot save works correctly in home screen, chrome, antutu,
youtube

Change-Id: Ib42ec810ae15933aa307923040bd88aa0017d336

M android/android-emugl/host/include/OpenGLESDispatch/gles2_extensions_functions.h
M android/android-emugl/host/libs/Translator/EGL/EglImp.cpp
M android/android-emugl/host/libs/Translator/GLES_CM/GLEScmImp.cpp
M android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp
M android/android-emugl/host/libs/Translator/GLcommon/SaveableTexture.cpp
M android/android-emugl/host/libs/Translator/include/GLcommon/SaveableTexture.h
M android/android-emugl/host/libs/Translator/include/GLcommon/TranslatorIfaces.h
M android/android-emugl/host/libs/libOpenGLESDispatch/gles2_extensions.entries
M android/third_party/tinyepoxy/tinyepoxy.cpp

https://android-review.googlesource.com/856470

https://goto.google.com/android-sha1/6286c8ca1fd2fe7d264500522c1b18a63effba2e

ap...@google.com <ap...@google.com> #45Dec 20, 2018 07:47PM

Project: platform/external/qemu
Branch: emu-3.0-release

commit 37caaa4390da7431248dce42e06391634e70efb9
Author: Lingfeng Yang <lfy@google.com>
Date: Thu Dec 20 08:46:02 2018

[snapshot][gl] Always read some formats with glReadPixels

bug: 120951634

This CL makes texture images with the following setup:

- GL_TEXTURE_2D
- level 0
- GL_RGB(A)
- GL_UNSIGNED_BYTE

to be read with glReadPixels.

This is meant to work around a macOS kernel panic.

Test: V1 Benchmark Pro texture save does not cause kernel panic on Mac
Test: Snapshot save works correctly in home screen, chrome, antutu,
youtube

Change-Id: Ib42ec810ae15933aa307923040bd88aa0017d336
(cherry picked from commit 6286c8ca1fd2fe7d264500522c1b18a63effba2e)

M android/android-emugl/host/include/OpenGLESDispatch/gles2_extensions_functions.h
M android/android-emugl/host/libs/Translator/EGL/EglImp.cpp
M android/android-emugl/host/libs/Translator/GLES_CM/GLEScmImp.cpp
M android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp
M android/android-emugl/host/libs/Translator/GLcommon/SaveableTexture.cpp
M android/android-emugl/host/libs/Translator/include/GLcommon/SaveableTexture.h
M android/android-emugl/host/libs/Translator/include/GLcommon/TranslatorIfaces.h
M android/android-emugl/host/libs/libOpenGLESDispatch/gles2_extensions.entries
M android/third_party/tinyepoxy/tinyepoxy.cpp

https://android-review.googlesource.com/856472

https://goto.google.com/android-sha1/37caaa4390da7431248dce42e06391634e70efb9

na...@gmail.com <na...@gmail.com> #46Dec 20, 2018 07:50PM

I've successfully exited the emulator in #43 without issue, several times. I'll keep testing with that build. The fix looks promising!

lf...@google.com <lf...@google.com> #47Dec 21, 2018 03:45AM

Glad it's working! We've now released it more widely as 28.0.22 canary.

lf...@google.com <lf...@google.com> #48Dec 21, 2018 07:57PM

Marked as fixed.

Now released to stable channel.

ng...@gmail.com <ng...@gmail.com> #49Dec 27, 2018 07:07AM

https://issuetracker.google.com/issues/120951634

ar...@gmail.com <ar...@gmail.com> #50Feb 20, 2019 05:40PM

Mi A2

ap...@google.com <ap...@google.com> #51Mar 2, 2019 12:44AM

Project: platform/external/qemu
Branch: emu-3.1-release

commit 5d76ed5de6661eaa0e1836f67ea46d5afc115599
Author: Lingfeng Yang <lfy@google.com>
Date: Thu Dec 20 08:46:02 2018

[snapshot][gl] Always read some formats with glReadPixels

bug: 120951634

This CL makes texture images with the following setup:

- GL_TEXTURE_2D
- level 0
- GL_RGB(A)
- GL_UNSIGNED_BYTE

to be read with glReadPixels.

This is meant to work around a macOS kernel panic.

Test: V1 Benchmark Pro texture save does not cause kernel panic on Mac
Test: Snapshot save works correctly in home screen, chrome, antutu,
youtube

Change-Id: Ib42ec810ae15933aa307923040bd88aa0017d336
(cherry picked from commit 6286c8ca1fd2fe7d264500522c1b18a63effba2e)

M android/android-emugl/host/include/OpenGLESDispatch/gles2_extensions_functions.h
M android/android-emugl/host/libs/Translator/EGL/EglImp.cpp
M android/android-emugl/host/libs/Translator/GLES_CM/GLEScmImp.cpp
M android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp
M android/android-emugl/host/libs/Translator/GLcommon/SaveableTexture.cpp
M android/android-emugl/host/libs/Translator/include/GLcommon/SaveableTexture.h
M android/android-emugl/host/libs/Translator/include/GLcommon/TranslatorIfaces.h
M android/android-emugl/host/libs/libOpenGLESDispatch/gles2_extensions.entries
M android/third_party/tinyepoxy/tinyepoxy.cpp

https://android-review.googlesource.com/917019

https://goto.google.com/android-sha1/5d76ed5de6661eaa0e1836f67ea46d5afc115599

ap...@google.com <ap...@google.com> #52Mar 2, 2019 12:44AM

Project: platform/external/qemu
Branch: emu-3.1-release

commit 0b7f6cad73163e70d65d9e6ac727e47642d0e98d
Author: Lingfeng Yang <lfy@google.com>
Date: Fri Dec 14 09:07:53 2018

[snapshot] Eagerly tear down RAM mappings on exit

bug: 120951634

Some users are experiencing kernel panic on snapshot save with
file-backed RAM when -gpu host is also enabled; the graphics driver
seems to cause a page fault in the kernel when saving on exit.

Speculative fix: Eagerly tear down RAM mappings.

+ a subtlety: Don't run ram block removal notifiers when exiting;
it's possible they were registered to a virtual device that has also
been torn down, possibly segfaulting in user space.

Change-Id: Iee1567dac2c997f1c89a573cb2e5a7369a1399ad
(cherry picked from commit 6241395508b6c846566fd423d83cd5ec607590ab)

M exec.c
M hw/i386/pc.c
M hw/i386/pc_piix.c
M include/exec/memory.h
M include/exec/ram_addr.h
M include/hw/boards.h
M include/hw/i386/pc.h
M memory.c
M vl.c

https://android-review.googlesource.com/917017

https://goto.google.com/android-sha1/0b7f6cad73163e70d65d9e6ac727e47642d0e98d

fo...@gmail.com <fo...@gmail.com> #53Aug 16, 2019 10:31AM

Chibai Chineses name

Issue 120951634

Description

Issue summary

Comments

uc...@google.com <uc...@google.com> #2Dec 13, 2018 11:04PM

de...@googlemail.com <de...@googlemail.com> #3Dec 14, 2018 12:02AM

lf...@google.com <lf...@google.com> #4Dec 14, 2018 12:04AM

sa...@gmail.com <sa...@gmail.com> #5Dec 14, 2018 03:36AM

lf...@google.com <lf...@google.com> #6Dec 14, 2018 03:59AM

ak...@gmail.com <ak...@gmail.com> #7Dec 14, 2018 01:23PM

lf...@google.com <lf...@google.com> #8Dec 14, 2018 01:53PM

ak...@gmail.com <ak...@gmail.com> #9Dec 14, 2018 02:15PM

ak...@gmail.com <ak...@gmail.com> #10Dec 14, 2018 02:28PM

lf...@google.com <lf...@google.com> #11Dec 14, 2018 02:41PM

lf...@google.com <lf...@google.com> #12Dec 14, 2018 02:43PM

ak...@gmail.com <ak...@gmail.com> #13Dec 14, 2018 03:14PM

lf...@google.com <lf...@google.com> #14Dec 14, 2018 03:27PM

ak...@gmail.com <ak...@gmail.com> #15Dec 14, 2018 03:28PM

lf...@google.com <lf...@google.com> #16Dec 14, 2018 05:41PM

lf...@google.com <lf...@google.com> #17Dec 14, 2018 06:14PM

ap...@google.com <ap...@google.com> #18Dec 14, 2018 06:54PM

ap...@google.com <ap...@google.com> #19Dec 14, 2018 06:55PM

lf...@google.com <lf...@google.com> Dec 14, 2018 10:19PM

lf...@google.com <lf...@google.com> #20Dec 14, 2018 11:37PM

ur...@gmail.com <ur...@gmail.com> #21Dec 15, 2018 07:34AM

lf...@google.com <lf...@google.com> #22Dec 17, 2018 03:30PM

ma...@gmail.com <ma...@gmail.com> #23Dec 18, 2018 11:42PM

sa...@gmail.com <sa...@gmail.com> #24Dec 19, 2018 03:07AM

lf...@google.com <lf...@google.com> #25Dec 19, 2018 03:17AM

ma...@gmail.com <ma...@gmail.com> #26Dec 19, 2018 04:17AM

lf...@google.com <lf...@google.com> #27Dec 19, 2018 04:20AM

ma...@gmail.com <ma...@gmail.com> #28Dec 19, 2018 04:27AM

lf...@google.com <lf...@google.com> #29Dec 19, 2018 04:29AM

ma...@gmail.com <ma...@gmail.com> #30Dec 19, 2018 04:36AM

na...@gmail.com <na...@gmail.com> #31Dec 19, 2018 02:34PM

yo...@gmail.com <yo...@gmail.com> #32Dec 19, 2018 02:39PM

lf...@google.com <lf...@google.com> #33Dec 19, 2018 03:43PM

na...@gmail.com <na...@gmail.com> #34Dec 19, 2018 04:02PM

lf...@google.com <lf...@google.com> #35Dec 19, 2018 04:46PM

lf...@google.com <lf...@google.com> #36Dec 19, 2018 08:56PM

lf...@google.com <lf...@google.com> #37Dec 20, 2018 03:22AM

na...@gmail.com <na...@gmail.com> #38Dec 20, 2018 01:53PM

na...@gmail.com <na...@gmail.com> #39Dec 20, 2018 02:06PM

na...@gmail.com <na...@gmail.com> #40Dec 20, 2018 02:13PM

lf...@google.com <lf...@google.com> #41Dec 20, 2018 06:12PM

lf...@google.com <lf...@google.com> #42Dec 20, 2018 06:14PM

lf...@google.com <lf...@google.com> #43Dec 20, 2018 06:54PM

ap...@google.com <ap...@google.com> #44Dec 20, 2018 07:33PM

ap...@google.com <ap...@google.com> #45Dec 20, 2018 07:47PM

na...@gmail.com <na...@gmail.com> #46Dec 20, 2018 07:50PM

lf...@google.com <lf...@google.com> #47Dec 21, 2018 03:45AM

lf...@google.com <lf...@google.com> #48Dec 21, 2018 07:57PM

ng...@gmail.com <ng...@gmail.com> #49Dec 27, 2018 07:07AM

ar...@gmail.com <ar...@gmail.com> #50Feb 20, 2019 05:40PM

ap...@google.com <ap...@google.com> #51Mar 2, 2019 12:44AM

ap...@google.com <ap...@google.com> #52Mar 2, 2019 12:44AM

fo...@gmail.com <fo...@gmail.com> #53Aug 16, 2019 10:31AM

bs...@gmail.com <bs...@gmail.com> #54May 12, 2020 06:06PM

Add comment

Issue metadata