ConcurrentCopy GC cause Native Crash happens on Android 13 and Android 12 devices increase rapidly since 12th Sep. [301833859]

Fixed

Bug

[AOSP] assigned

Status Update

No update yet.

Description

qi...@trip.com

created issue #1

Sep 25, 2023 07:23AM

PLEASE READ:

On Android 12 & 13 devices, our app received a GC-related native crash since 12th Sep(it's very strange that we full rollout on 9th Sep, and the crash we received on 10th and 11th is quite normal, 12th increased rapidly and kept the trend afterwards). The logs seemed to be a GC-related crash, and I hope the google team can help to figure out. We tried to reproduce it offline, but failed. There are two similar error stack issues, please see the attachments.

For addition, I noticed that "android system update" in google play store updated on Sep 8, 2023, dont know does it related or not.

Looking forward for your reply, this issue is quite urgent.

Thanks

deleted

Restricted

0 B

deleted

Restricted

0 B

Comments

vi...@google.com <vi...@google.com> #2Sep 25, 2023 12:54PM

Assigned to vi...@google.com.

Could you please provide a sample project that reproduces your issue?

You are right about restoring the request codes, but under normal usage of the API, that should not cause this failure.

qi...@trip.com <qi...@trip.com> #3Sep 25, 2023 01:36PM

Unfortunately I'm unable to reproduce the issue neither in a sample nor in the real project.

These crashes are received from the users of production app. I'll try to add more logs for better understanding what users do before the crash.

qi...@trip.com <qi...@trip.com> #4Sep 26, 2023 11:07AM

Project: platform/frameworks/support
Branch: androidx-master-dev

commit b2a3eca325760e3686411e619048e2bc52befda8
Author: Jeremy Woods <jbwoods@google.com>
Date: Wed Sep 16 16:55:46 2020

Ensure request code start is consistent after restore

When the ActivityResultRegistry is recreated due to either config change
or process death, the next request code is started at the number of
current request codes without including the initial offset.

We should make sure to include the initial offset on restore to remain
consistent.

Relnote: "When restoring the ActivityResultRegistry, the request codes
are now properly started from their previous value instead of the value
of the number of existing request codes."
Test: all ActivityResultRegistryTest pass
Bug: 168374000

Change-Id: I463711fe19233877f1b534e4a7a3325fcfd8f983

M activity/activity/src/main/java/androidx/activity/result/ActivityResultRegistry.java

https://android-review.googlesource.com/1428862

vi...@google.com <vi...@google.com> #5Sep 27, 2023 12:40PM

This change ensures the request code matches, if your issue is not solved and you find a way to reproduce it, please file another bug.

qi...@gmail.com <qi...@gmail.com> #6Sep 27, 2023 12:53PM

Comment has been deleted.

Message last modified on Sep 27, 2023 12:54PM

qi...@trip.com <qi...@trip.com> #7Sep 27, 2023 12:54PM

It's because we can't reproduce it that we submitted this issue, but you can't analyze it any further without reproducing it, if you can help to analysis a little bit in-depth, it would be very appreciated.

vi...@google.com <vi...@google.com> #8Sep 28, 2023 03:37PM

We could not find anything relevant in the shared log please provide a fresh log details to proceed further, thanks .

qi...@gmail.com <qi...@gmail.com> #9Sep 30, 2023 08:06AM

Comment has been deleted.

Message last modified on Sep 30, 2023 08:06AM

deleted

Restricted

0 B

qi...@trip.com <qi...@trip.com> #10Sep 30, 2023 08:07AM

Sure, I'v uploaded 6 fresh log details, plz see the attachment, thanks

deleted

Restricted

0 B

vi...@google.com <vi...@google.com> #11Oct 1, 2023 12:16AM

We have shared this with our product and engineering team and will update this issue with more information as it becomes available.

qi...@trip.com <qi...@trip.com> #12Oct 6, 2023 01:21PM

I'd like to know if there's any further progress on this issue please?

ng...@google.com <ng...@google.com> #13Oct 9, 2023 09:02AM

Hi trip.com team. We're still investigating but unfortunately we haven't found the cause yet.

Do you have any native code (.so files) in your APKs? Or is your app only in Java?

Are you using an external Android SDK? And/or an obfuscator?

qi...@gmail.com <qi...@gmail.com> #14Oct 9, 2023 09:33AM

Comment has been deleted.

Message last modified on Oct 9, 2023 09:34AM

qi...@trip.com <qi...@trip.com> #15Oct 9, 2023 09:34AM

Hi Google Team, Yes, We do have native code in our APK and also third-party sdks.

Rephrase：we noticed that the this issue increased on 24th Aug for first time, before that the rate of this crash occurrence is rare, even after 24th Aug increased, the crash rate of this issue was acceptable, but on 12th Sep increased rapidly. We are sure that we did not release or update any configurations of features in our app that leads to this crash.(We got some information from other companies, for which release app to global user on Google Play Store, that they also faced the similar issue since end of Aug)

For next step we gonna to try is open the GWP-Asan and phase release to user, to collect memory allocating information, the information will be provided once we get from online user devices.

Please help to continuously analysis this issue, it's been bothering us for a long time and urgent!

Thanks.

ng...@google.com <ng...@google.com> #16Oct 9, 2023 09:38AM

I suggest working with the third party SDKs. Some of them have notoriously been fragile in the presence of OTAs and system updates.

qi...@gmail.com <qi...@gmail.com> #17Oct 9, 2023 10:23AM

Comment has been deleted.

Message last modified on Oct 9, 2023 10:24AM

qi...@trip.com <qi...@trip.com> #18Oct 9, 2023 10:26AM

Yes, we are working on it, and helping from your side is very important, to narrow the scope of investigating, if we can get some specifics leads to this issue would be great!

qi...@trip.com <qi...@trip.com> #19Oct 10, 2023 01:35PM

Hi Google Team, we reproduced this crash via kind of monkey test, so can not provide the steps to reproduce.

Please see the log file and tombstone file.

deleted

Restricted

0 B

deleted

Restricted

0 B

qi...@trip.com <qi...@trip.com> #20Oct 11, 2023 07:59AM

Hi Google Team,

I've uploaded three more tombstone files, please help to analysis in-depth, thanks!

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

de...@similarweb.com <de...@similarweb.com> #21Oct 12, 2023 02:25PM

Our app is facing the exact same issue, we see a huge spike in native crashes of 2 kinds on Sep 12 2023 (for two different apps, which means it's not tied to the release of a specific app version as these are not synchronised). Both crashes only happen on Android 13 and 14. One of the apps has an external SDK with native code, while the other one does not (Kotlin code only).
Crash 1:
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 13414 >>> com.embeepay.mpm <<<

backtrace:
#00 pc 0x000000000020598c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::Copy(art::Thread*, art::mirror::Object*, art::mirror::Object*, art::MemberOffset)+68)
#01 pc 0x0000000000207d58 /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&)+984)
#02 pc 0x0000000000257394 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::CopyingPhase()+556)
#03 pc 0x000000000024475c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+468)
#04 pc 0x00000000003bde5c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#05 pc 0x00000000003bb8c0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#06 pc 0x00000000003bb114 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#07 pc 0x00000000003baf98 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#08 pc 0x0000000000393830 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#09 pc 0x0000000000316514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#10 pc 0x00000000005e8698 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#11 pc 0x00000000005bb5dc /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#12 pc 0x00000000003fe098 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#13 pc 0x00000000003605a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#14 pc 0x000000000034b930 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#15 pc 0x00000000004f3e38 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#16 pc 0x00000000001018e4 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+344)
#17 pc 0x000000000009affc /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

Crash 2:
Invalid reference: ref=0x14790fe8 referenced from: object=0x144810f0 offset= 32
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 19196 >>> com.embeepay.mpm <<<

backtrace:
#00 pc 0x00000000000531f4 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164)
#01 pc 0x00000000007910cc /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+1380)
#02 pc 0x00000000000357d0 /apex/com.google.mainline.primary.libs@340817060/lib64/

libbase.so/5cbcb48dc4a1ceb71422e6e07e9f12bb952b969b1d4789234c07c3b6241482a6f3a09733800aa23496f3dcd787dd1ded12735fe6dc5b3049d050d96b769bc049/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+80)
#03 pc 0x0000000000034d58 /apex/com.google.mainline.primary.libs@340817060/lib64/

libbase.so/5cbcb48dc4a1ceb71422e6e07e9f12bb952b969b1d4789234c07c3b6241482a6f3a09733800aa23496f3dcd787dd1ded12735fe6dc5b3049d050d96b769bc049/libbase.so (android::base::LogMessage::~LogMessage()+352)
#04 pc 0x0000000000207724 /apex/com.android.art/lib64/libart.so (art::gc::Verification::LogHeapCorruption(art::ObjPtr<art::mirror::Object>, art::MemberOffset, art::mirror::Object*, bool) const+1976)
#05 pc 0x000000000020a3f0 /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&)+10864)
#06 pc 0x0000000000257394 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::CopyingPhase()+556)
#07 pc 0x000000000024475c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+468)
#08 pc 0x00000000003bde5c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#09 pc 0x00000000003bb8c0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#10 pc 0x00000000003bb114 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#11 pc 0x00000000003baf98 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#12 pc 0x0000000000393830 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#13 pc 0x0000000000303514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#14 pc 0x00000000005d5038 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#15 pc 0x00000000005a7f7c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#16 pc 0x00000000003ea7c8 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#17 pc 0x00000000003605a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#18 pc 0x000000000034b930 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#19 pc 0x00000000004f3e38 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#20 pc 0x00000000000c226c /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+204)
#21 pc 0x0000000000054a30 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

Is there any possibility this could be caused by some external condition affecting multiple apps?

deleted

Restricted

0 B

deleted

Restricted

0 B

qi...@trip.com <qi...@trip.com> #22Oct 13, 2023 10:21AM

Hi Google Team,

This issue seems to be a common problem, other apps(published in Google play store) in our company have been affected too, and also affected the app in other companies(please see #21).

Appreciate for have been helping investigate this issue, and looking forward to your replies.

Thanks.

qi...@trip.com <qi...@trip.com> #23Oct 16, 2023 04:54AM

Hi Google Team,

We have reproduced this issue via MTE device by running kind of autotest in an other app of our company. Please see details in attachment!

In terms of this issue was triggered when executing the flutter engine code(libfutter.so related) as the tombstone file has shown, the Flutter engine version of that app is 1.22.6.

Looking forward to your replies.

Thanks.

tombstone_00-MTE

3.8 MB

Download

ng...@google.com <ng...@google.com> #24Oct 16, 2023 10:58AM

The call coming from flutter, can you file a bug on the flutter engine bug tracker?

qi...@gmail.com <qi...@gmail.com> #25Oct 16, 2023 11:05AM

Comment has been deleted.

Message last modified on Oct 16, 2023 11:06AM

qi...@trip.com <qi...@trip.com> #26Oct 16, 2023 11:07AM

Hi Google Team,

Do you confirmed that this issue I'v reported to you(not only about #23 I'v updated) is related to Flutter engine from your side?

BTW, the comment in #21 has shown that this issue occurred in the app with only kotlin code, which could not explain this bug is related to Flutter engine!

Please help to investigate a little bit in-depth and looking forward to your replies.

Thanks.

de...@similarweb.com <de...@similarweb.com> #27Oct 16, 2023 11:10AM

Hi Google Team,

I am confirming that apps listed in #21 are not using Flutter. We don't have tombstones as we haven't managed to reproduce the issue, we only have reports from the Play Store.

Thanks.

ng...@google.com <ng...@google.com> #28Oct 16, 2023 11:21AM

Hard to say what is the issue in #21 without repro. Even Java code can tamper with native memory with Unsafe.

qi...@trip.com <qi...@trip.com> #29Oct 16, 2023 11:29AM

But crash increased rapidly on the same day(12th Sep), and it's obviously that both of us are not know each other, and no business cross. It‘s kind of common issue.

Therefore, please raise the priority of this issue, thanks.

de...@similarweb.com <de...@similarweb.com> #30Oct 16, 2023 05:58PM

Hi Google Team,

I noticed a debug version of the app crash today and was able to capture tombstone via logcat:
Occurrence 1:
2023-10-16 11:48:45.910 24280-24289 libc pid-24280 A Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 24289 (HeapTaskDaemon), pid 24280 (om.embeepay.mpm)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A Build fingerprint: 'google/oriole/oriole:14/UP1A.231005.007/10754064:user/release-keys'
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A Revision: 'MP1.0'
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A ABI: 'arm64'
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A Timestamp: 2023-10-16 11:48:46.068712459+0300
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A Process uptime: 10215s
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A Cmdline: com.embeepay.mpm
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A pid: 24280, tid: 24289, name: HeapTaskDaemon >>> com.embeepay.mpm <<<
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A uid: 11336
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A Abort message: ' Invalid reference: ref=0x174009a0 referenced from: object=0x173c0c70 offset= 116
obj=0x173c0c70 klass=0x708babf0(java.lang.Class<java.lang.String[]>) length=36 space=main space (region space) 0x12c00000-0x32c00000 card=111 adjacent_ram=10000000708d09d0 0000000000000000 0000000316ec4188 00000000172000c0 |00000000708babf0 0000000000000024 173c098870d61198 70d6119800000000
obj->GetMarkBit()=0
obj->GetReadBarrierState()=0
Region containing obj:
Region[287]=0x173c0000-0x173eb020-0x17400000 state=RegionStateAllocated type=RegionTypeUnevacFromSpace objects_allocated=3463 alloc_time=380 live_bytes=18446744073709551615 is_newly_allocated=false is_a_tlab=false thread=0x0
region_space_bitmap_->Test(obj)=true
ref=0x174009a0 klass=0x0 <invalid address> space=main space (region space) 0x12c00000-0x32c00000 card=0 adjacent_ram=0000000000000000 0000000000000000 0000000000000000 0000000000000000 |0000000000000000 0000000000000000 0000000000000000 0000000000000000
ref->GetMarkBit()=0
ref->GetReadBarrierState()=0
Region containing ref:
Region[288]=0x17400000-0x17400000-0x17440000 state=RegionStateFree type=RegionTypeNone objects_allocated=0 alloc_time=0 live_bytes=18446744073709551615 is_newly_allocated=false is_a_tlab=false thread=0x0
region_space_bitmap_->Test(ref)=false'
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A x0 0000000000000000 x1 0000000000005ee1 x2 0000000000000006 x3 00000072768c1120
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A x4 00000000676f6c62 x5 00000000676f6c62 x6 00000000676f6c62 x7 b4000073bd214516
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A x8 00000000000000f0 x9 00000075a2674050 x10 0000000000000001 x11 00000075a26bdb60
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A x12 000000000000eedc x13 000000007fffffff x14 0000000003dd1760 x15 000006cb54f53e4e
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A x16 00000075a2729cf8 x17 00000075a2706470 x18 0000007275b16000 x19 0000000000005ed8
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A x20 0000000000005ee1 x21 00000000ffffffff x22 000000000000006e x23 0000000000000001
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A x24 00000072e8abfab4 x25 0000000000000002 x26 0000000000000001 x27 000000000000006e
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A x28 00000072e9616000 x29 00000072768c11a0
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A lr 00000075a26ae178 sp 00000072768c1100 pc 00000075a26ae1a4 pst 0000000000001000
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A 37 total frames
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A backtrace:
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #00 pc 000000000005c1a4 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #01 pc 00000000007845a0 /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+904) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #02 pc 00000000000357d0 /apex/com.google.mainline.primary.libs@340941000/lib64/

libbase.so/75d3253827fcfd7a8d7b02ad45991611ec4ca424c0278e13e8acfad4d14e597a3ecff6c0caa2b785c73838528ee6e9c2b313240ff895f50ee39b1d7bc10f390a/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+80) (BuildId: 6f67f69ff36b970d0b831cfdab3b578d)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #03 pc 0000000000034d58 /apex/com.google.mainline.primary.libs@340941000/lib64/

libbase.so/75d3253827fcfd7a8d7b02ad45991611ec4ca424c0278e13e8acfad4d14e597a3ecff6c0caa2b785c73838528ee6e9c2b313240ff895f50ee39b1d7bc10f390a/libbase.so (android::base::LogMessage::~LogMessage()+352) (BuildId: 6f67f69ff36b970d0b831cfdab3b578d)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #04 pc 0000000000207734 /apex/com.android.art/lib64/libart.so (art::gc::Verification::LogHeapCorruption(art::ObjPtr<art::mirror::Object>, art::MemberOffset, art::mirror::Object*, bool) const+2016) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #05 pc 000000000020a3bc /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&)+10588) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #06 pc 00000000002b3b20 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::CopyingPhase()+1044) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #07 pc 00000000002a1210 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+468) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #08 pc 00000000003aa57c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+312) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #09 pc 00000000003a6500 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+540) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #10 pc 00000000003a5d38 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #11 pc 00000000003a5bbc /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #12 pc 00000000003a2a2c /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+72) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #13 pc 000000000034dc30 /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+144) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #14 pc 00000000003371a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #15 pc 0000000000519790 /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, bool, art::JValue*)+1976) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #16 pc 00000000004a2e3c /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext*)+960) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #17 pc 00000000003503d8 /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #18 pc 000000000002bae0 /apex/com.android.art/javalib/core-libart.jar (java.lang.Daemons$HeapTaskDaemon.runInternal+0)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #19 pc 000000000036c700 /apex/com.android.art/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.__uniq.112435418011751916792819755956732575238.llvm.3816585244953842104)+232) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #20 pc 000000000051a45c /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, bool, art::JValue*)+5252) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #21 pc 00000000004a2e3c /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext*)+960) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.557 25149-25149 DEBUG pid-25149 A #22 pc 00000000003503d8 /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #23 pc 000000000002ae0c /apex/com.android.art/javalib/core-libart.jar (java.lang.Daemons$Daemon.run+0)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #24 pc 000000000036c700 /apex/com.android.art/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.__uniq.112435418011751916792819755956732575238.llvm.3816585244953842104)+232) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #25 pc 000000000051a45c /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, bool, art::JValue*)+5252) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #26 pc 00000000004a3ce8 /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext*)+4716) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #27 pc 00000000003503d8 /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #28 pc 000000000010ee0c /apex/com.android.art/javalib/core-oj.jar (java.lang.Thread.run+0)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #29 pc 000000000036c700 /apex/com.android.art/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.__uniq.112435418011751916792819755956732575238.llvm.3816585244953842104)+232) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #30 pc 000000000036bff8 /apex/com.android.art/lib64/libart.so (artQuickToInterpreterBridge+964) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #31 pc 000000000034dd68 /apex/com.android.art/lib64/libart.so (art_quick_to_interpreter_bridge+88) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #32 pc 00000000003371a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #33 pc 000000000023ea64 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #34 pc 000000000054436c /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1600) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #35 pc 00000000000c9ccc /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+204) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
2023-10-16 11:48:46.558 25149-25149 DEBUG pid-25149 A #36 pc 000000000005db00 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 19c32900d9d702c303d2b4164fbba76c)

Occurrence 2:
23-10-16 17:46:07.656 25789-25801 libc pid-25789 A Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 25801 (HeapTaskDaemon), pid 25789 (

com.embee.uk)
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Build fingerprint: 'google/oriole/oriole:14/UP1A.231005.007/10754064:user/release-keys'
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Revision: 'MP1.0'
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A ABI: 'arm64'
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Timestamp: 2023-10-16 17:46:07.913669634+0300
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Process uptime: 440s
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Cmdline:

com.embee.uk
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A pid: 25789, tid: 25801, name: HeapTaskDaemon >>>

com.embee.uk <<<
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A uid: 10345
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Abort message: 'GC tried to mark invalid reference 0x151dd120
ref=0x151dd120 klass=0x0 <invalid address> space=main space (region space) 0x12c00000-0x32c00000 card=0 adjacent_ram=0000000000000000 0000000000000000 0000000000000000 0000000000000000 |0000000000000000 0000000000000000 0000000000000000 0000000000000000
holder=0x15181710 klass=0x708babf0(java.lang.Class<java.lang.String[]>) length=48 space=main space (region space) 0x12c00000-0x32c00000 card=111 adjacent_ram=0000000000000000 0000000000000000 0000000000000000 0000000000000000 |00000000708babf0 0000000000000030 1518107070d61198 70d6119800000000
field_offset=188 reference addr adjacent_ram=70d6119800000000 0000000015180480 1518038870d61198 70d6119800000000 |00000000151dd120 151de08070d61198 708981e000000000 0000001000000000 0xb40000747d20c200 main space (region space) 0x12c00000-0x32c00000
0xb40000747d20c4e8 region space live bitmap[begin=0x12c00000,end=0x32c00000]
0xb40000747d20c4e8 region space live bitmap[begin=0x12c00000,end=0x32c00000]
0xb4000073fd20b610 SpaceTypeImageSpace begin=0x70703000,end=0x70fb88d8,size=8918KB,name="/data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.art"]
0xb4000073fd20b698 imagespace /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.art live-bitmap 0[begin=0x70703000,end=0x70fb8c00]
0xb4000073fd20b698 imagespace /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.art live-bitmap 0[begin=0x70703000,end=0x70fb8c00]
0xb4000073fd20add0 SpaceTypeImageSpace begin=0x72a4a000,end=0x72d319f0,size=2974KB,name="/data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework-adservices.art"]
0xb4000073fd20ae58 imagespace /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework-adservices.art live-bitmap 1[begin=0x72a4a000,end=0x72d31c00]
0xb4000073fd20ae58 imagespace /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework-adservices.art live-bitmap 1[begin=0x72a4a000,end=0x72d31c00]
0xb40000746d20ef40 SpaceTypeZygoteSpace begin=0x730e3000,end=0x73702000,size=6268KB,name="Zygote space"]
0xb40000746d20efd0 allocspace zygote / non moving space live-bitmap 0[begin=0x730e3000,end=0x73702000]
0xb40000746d20f050 allocspace zygote / non moving space live-bitmap 0[begin=0x730e3000,end=0x73702000]
0xb40000742d20e170 SpaceTypeMallocSpace begin=0x73702000,end=0x73728000,limit=0x770e3000,size=152KB,capacity=57MB,non_growth_limit_capacity=57MB,name="non moving space"]
0xb40000742d20e200 allocspace non moving space live-bitmap 1[begin=0x73702000,end=0x770e3000]
0xb40000742d20e280 allocspace non moving space live-bitmap 1[begin=0x73702000,end=0x770e3000]
0xb4000073fd250bb0 SpaceTypeImageSpace begin=0x9f0e3000,end=0x9f1a88c8,size=790KB,name="/data/app/~~EkgDN7qzK9d2R3Xi-MHc6g==/com.embee.uk-DKkN2lE_hTB8UCAkT3kc7w==/oat/arm64/base.art"]
0xb4000073fd250c38 imagespace /data/app/~~EkgDN7qzK9d2R3Xi-MHc6g==/com.embee.uk-DKkN2lE_hTB8UCAkT3kc7w==/oat/arm64/base.art live-bitmap 2[begin=0x9f0e3000,end=0x9f1a8c00]
0xb4000073fd250c38 imagespace /data/app/~~EkgDN7qzK9d2R3Xi-MHc6g==/com.embee.uk-DKkN2lE_hTB8UCAkT3kc7w==/oat/arm64/base.art live-bitmap 2[begin=0x9f0e3000,end=0x9f1a8c00]
0xb4000073fd22a3b0 SpaceTypeImageSpace begin=0x9f2a0000,end=0x9f2a0308,size=776B,name="/data/misc/apexdata/com.android.art/dalvik-cache/arm64/system@framework@com.android.location.provider.jar@classes.art"]
0xb4000073fd22a438 imagespace /data/misc/apexdata/com.android.art/dalvik-cache/arm64/system@framework@com.android.location.provider.jar@classes.art live-bitmap 3[begin=0x9f2a0000,end=0x9f2a0400]
0xb4000073fd22a438 imagespace /data/misc/apexdata/com.android.art/dalvik-cache/arm64/system@framework@com.android.location.provider.jar@classes.art live-bitmap 3[begin=0x9f2a0000,end=0x9f2a0400]
0xb40000746d2084e0 free list large object space - begin: 0x770e3000 end: 0x970e3000
Large object at address: 0x770e3000 of length 16384 bytes
Large object at address: 0x770e7000 of length 20480 bytes
Free block at address: 0x770ec000 of length 4096 bytes
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Large object at address: 0x770ed000 of length 49152 bytes
Large object at address: 0x770f9000 of length 49152 bytes
Large object at address: 0x77105000 of length 65536 bytes
Large object at address: 0x77115000 of length 32768 bytes
Large object at address: 0x7711d000 of length 20480 bytes
Large object at address: 0x77122000 of length 20480 bytes
Free block at address: 0x77127000 of length 4096 bytes
Large object at address: 0x77128000 of length 24576 bytes
Large object at address: 0x7712e000 of length 36864 bytes
Large object at address: 0x77137000 of length 20480 bytes
Free block at address: 0x7713c000 of length 4096 bytes
Large object at address: 0x7713d000 of length 16384 bytes
Large object at address: 0x77141000 of length 20480 bytes
Large object at address: 0x77146000 of length 20480 bytes
Large object at address: 0x7714b000 of length 20480 bytes
Large object at address: 0x77150000 of length 20480 bytes
Large object at address: 0x77155000 of length 20480 bytes
Large object at address: 0x7715a000 of length 20480 bytes
Free block at address: 0x7715f000 of length 8192 bytes
Large object at address: 0x77161000 of length 20480 bytes
Free block at address: 0x77166000 of length 8192 bytes
Large object at address: 0x77168000 of length 28672 bytes
Large object at address: 0x7716f000 of length 20480 bytes
Large object at address: 0x77174000 of length 28672 bytes
Large object at address: 0x7717b000 of length 16384 bytes
Large object at address: 0x7717f000 of length 16384 bytes
Large object at address: 0x77183000 of length 16384 bytes
Large object at address: 0x77187000 of length 24576 bytes
Large object at address: 0x7718d000 of length 20480 bytes
Free block at address: 0x77192000 of length 8192 bytes
Large object at address: 0x77194000 of length 28672 bytes
Large object at address: 0x7719b000 of length 28672 bytes
Large object at address: 0x771a2000 of length 20480 bytes
Free block at address: 0x771a7000 of length 4096 bytes
Large object at address: 0x771a8000 of length 24576 bytes
Large object at address: 0x771ae000 of length 20480 bytes
Large object at address: 0x771b3000 of length 20480 bytes
Large object at address: 0x771b8000 of length 20480 bytes
Large object at address: 0x771bd000 of length 20480 bytes
Large object at address: 0x771c2000 of length 266240 bytes
Large object at address: 0x77203000 of length 32768 bytes
Large object at address: 0x7720b000 of length 20480 bytes
Free block at address: 0x77210000 of length 4096 bytes
Large object at address: 0x77211000 of length 24576 bytes
Free block at address: 0x77217000 of length 16384 bytes
Large object at address: 0x7721b000 of length 16384 bytes
Large object at address: 0x7721f000 of length 479232 bytes
Large object at address: 0x77294000 of length 16384 bytes
Large object at address: 0x77298000 of length 24576 bytes
Large object at address: 0x7729e000 of length 57344 bytes
Large object at address: 0x772ac000 of length 20480 bytes
Large object at address: 0x772b1000 of length 16384 bytes
Free block at address: 0x772b5000 of length 16384 bytes
Large object at address: 0x772b9000 of length 16384 bytes
Large object at address: 0x772bd000 of length 20480 bytes
Large object at address: 0x772c2000 of length 20480 bytes
Large object at address: 0x772c7000 of length 20480 bytes
Free block at address: 0x772cc000 of length 4096 bytes
Large object at address: 0x772cd000 of length 65536 bytes
Free block at address: 0x772dd000 of length 16384 bytes
Large object at address: 0x772e1000 of length 16384 bytes
Large object at address: 0x772e5000 of length 16384 bytes
Large object at address: 0x772e9000 of length 16384 bytes
Free block at address: 0x772ed000 of length 16384 bytes
Large object at address: 0x772f1000 of length 16384 bytes
Free block at address: 0x772f5000 of length 16384 bytes
Large object at address: 0x772f9000 of length 16384 bytes
Free block at address: 0x772fd000 of length 16384 bytes
Large object at address: 0x77301000 of length 16384 bytes
Free block at address: 0x77305000 of length 16384 bytes
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Large object at address: 0x77309000 of length 16384 bytes
Free block at address: 0x7730d000 of length 16384 bytes
Large object at address: 0x77311000 of length 16384 bytes
Free block at address: 0x77315000 of length 16384 bytes
Large object at address: 0x77319000 of length 16384 bytes
Free block at address: 0x7731d000 of length 16384 bytes
Large object at address: 0x77321000 of length 16384 bytes
Large object at address: 0x77325000 of length 20480 bytes
Large object at address: 0x7732a000 of length 20480 bytes
Large object at address: 0x7732f000 of length 20480 bytes
Large object at address: 0x77334000 of length 20480 bytes
Large object at address: 0x77339000 of length 20480 bytes
Large object at address: 0x7733e000 of length 20480 bytes
Large object at address: 0x77343000 of length 20480 bytes
Free block at address: 0x77348000 of length 4096 bytes
Large object at address: 0x77349000 of length 147456 bytes
Large object at address: 0x7736d000 of length 20480 bytes
Large object at address: 0x77372000 of length 20480 bytes
Large object at address: 0x77377000 of length 40960 bytes
Free block at address: 0x77381000 of length 16384 bytes
Large object at address: 0x77385000 of length 16384 bytes
Free block at address: 0x77389000 of length 16384 bytes
Large object at address: 0x7738d000 of length 16384 bytes
Large object at address: 0x77391000 of length 20480 bytes
Large object at address: 0x77396000 of length 20480 bytes
Free block at address: 0x7739b000 of length 8192 bytes
Large object at address: 0x7739d000 of length 49152 bytes
Large object at address: 0x773a9000 of length 20480 bytes
Free block at address: 0x773ae000 of length 12288 bytes
Large object at address: 0x773b1000 of length 28672 bytes
Large object at address: 0x773b8000 of length 16384 bytes
Large object at address: 0x773bc000 of length 69632 bytes
Large object at address: 0x773cd000 of length 69632 bytes
Large object at address: 0x773de000 of length 36864 bytes
Large object at address: 0x773e7000 of length 69632 bytes
Large object at address: 0x773f8000 of length 36864 bytes
Free block at address: 0x77401000 of length 1462272 bytes
Large object at address: 0x77566000 of length 20480 bytes
Large object at address: 0x7756b000 of length 20480 bytes
Large object at address: 0x77570000 of length 20480 bytes
Large object at address: 0x77575000 of length 69632 bytes
Large object at address: 0x77586000 of length 69632 bytes
Large object at address: 0x77597000 of length 36864 bytes
Large object at address: 0x775a0000 of length 69632 bytes
Large object at address: 0x775b1000 of length 36864 bytes
Large object at address: 0x775ba000 of length 24576 bytes
Large object at address: 0x775c0000 of length 49152 bytes
Free block at address: 0x775cc000 of length 12288 bytes
Large object at address: 0x775cf000 of length 20480 bytes
Large object at address: 0x775d4000 of length 20480 bytes
Large object at address: 0x775d9000 of length 20480 bytes
Large object at address: 0x775de000 of length 20480 bytes
Large object at address: 0x775e3000 of length 20480 bytes
Large object at address: 0x775e8000 of length 143360 bytes
Large object at address: 0x7760b000 of length 143360 bytes
Free block at address: 0x7762e000 of length 1863680 bytes
Large object at address: 0x777f5000 of length 20480 bytes
Large object at address: 0x777fa000 of length 94208 bytes
Large object at address: 0x77811000 of length 53248 bytes
Large object at address: 0x7781e000 of length 28672 bytes
Large object at address: 0x77825000 of length 53248 bytes
Large object at address: 0x77832000 of length 53248 bytes
Large object at address: 0x7783f000 of length 69632 bytes
Large object at address: 0x77850000 of length 36864 bytes
Large object at address: 0x77859000 of length 69632 bytes
Large object at address: 0x7786a000 of length 36864 bytes
Large object at address: 0x77873000 of length 20480 bytes
Large object at address: 0x77878000 of length 20480 bytes
Large object at address: 0x7787d000 of length 20480 bytes
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Large object at address: 0x77882000 of length 20480 bytes
Large object at address: 0x77887000 of length 20480 bytes
Large object at address: 0x7788c000 of length 20480 bytes
Large object at address: 0x77891000 of length 20480 bytes
Large object at address: 0x77896000 of length 20480 bytes
Large object at address: 0x7789b000 of length 20480 bytes
Large object at address: 0x778a0000 of length 20480 bytes
Large object at address: 0x778a5000 of length 20480 bytes
Large object at address: 0x778aa000 of length 20480 bytes
Large object at address: 0x778af000 of length 20480 bytes
Large object at address: 0x778b4000 of length 20480 bytes
Large object at address: 0x778b9000 of length 20480 bytes
Large object at address: 0x778be000 of length 20480 bytes
Free block at address: 0x778c3000 of length 4096 bytes
Large object at address: 0x778c4000 of length 20480 bytes
Large object at address: 0x778c9000 of length 20480 bytes
Large object at address: 0x778ce000 of length 20480 bytes
Large object at address: 0x778d3000 of length 20480 bytes
Large object at address: 0x778d8000 of length 69632 bytes
Large object at address: 0x778e9000 of length 36864 bytes
Large object at address: 0x778f2000 of length 69632 bytes
Large object at address: 0x77903000 of length 36864 bytes
Large object at address: 0x7790c000 of length 20480 bytes
Free block at address: 0x77911000 of length 8192 bytes
Large object at address: 0x77913000 of length 233472 bytes
Large object at address: 0x7794c000 of length 20480 bytes
Large object at address: 0x77951000 of length 20480 bytes
Large object at address: 0x77956000 of length 20480 bytes
Large object at address: 0x7795b000 of length 20480 bytes
Large object at address: 0x77960000 of length 20480 bytes
Large object at address: 0x77965000 of length 20480 bytes
Free block at address: 0x7796a000 of length 16384 bytes
Large object at address: 0x7796e000 of length 98304 bytes
Large object at address: 0x77986000 of length 98304 bytes
Large object at address: 0x7799e000 of length 69632 bytes
Large object at address: 0x779af000 of length 20480 bytes
Large object at address: 0x779b4000 of length 20480 bytes
Large object at address: 0x779b9000 of length 20480 bytes
Large object at address: 0x779be000 of length 20480 bytes
Large object at address: 0x779c3000 of length 69632 bytes
Large object at address: 0x779d4000 of length 36864 bytes
Large object at address: 0x779dd000 of length 20480 bytes
Large object at address: 0x779e2000 of length 20480 bytes
Large object at address: 0x779e7000 of length 20480 bytes
Large object at address: 0x779ec000 of length 20480 bytes
Large object at address: 0x779f1000 of length 20480 bytes
Large object at address: 0x779f6000 of length 20480 bytes
Large object at address: 0x779fb000 of length 20480 bytes
Large object at address: 0x77a00000 of length 20480 bytes
Large object at address: 0x77a05000 of length 36864 bytes
Free block at address: 0x77a0e000 of length 16384 bytes
Large object at address: 0x77a12000 of length 20480 bytes
Large object at address: 0x77a17000 of length 20480 bytes
Large object at address: 0x77a1c000 of length 20480 bytes
Large object at address: 0x77a21000 of length 20480 bytes
Large object at address: 0x77a26000 of length 20480 bytes
Large object at address: 0x77a2b000 of length 20480 bytes
Large object at address: 0x77a30000 of length 20480 bytes
Large object at address: 0x77a35000 of length 20480 bytes
Large object at address: 0x77a3a000 of length 20480 bytes
Large object at address: 0x77a3f000 of length 20480 bytes
Large object at address: 0x77a44000 of length 20480 bytes
Large object at address: 0x77a49000 of length 20480 bytes
Large object at address: 0x77a4e000 of length 20480 bytes
Free block at address: 0x77a53000 of length 4096 bytes
Large object at address: 0x77a54000 of length 20480 bytes
Large object at address: 0x77a59000 of length 20480 bytes
Large object at address: 0x77a5e000 of length 20480 bytes
Large object at address: 0x77a63000 of length 20480 bytes
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A Large object at address: 0x77a68000 of length 36864 bytes
Large object at address: 0x77a71000 of length 20480 bytes
Large object at address: 0x77a76000 of length 20480 bytes
Large object at address: 0x77a7b000 of length 20480 bytes
Large object at address: 0x77a80000 of length 20480 bytes
Large object at address: 0x77a85000 of length 20480 bytes
Large object at address: 0x77a8a000 of length 69632 bytes
Large object at address: 0x77a9b000 of length 36864 bytes
Large object at address: 0x77aa4000 of length 69632 bytes
Large object at address: 0x77ab5000 of length 36864 bytes
Large object at address: 0x77abe000 of length 69632 bytes
Large object at address: 0x77acf000 of length 36864 bytes
Large object at address: 0x77ad8000 of length 20480 bytes
Large object at address: 0x77add000 of length 69632 bytes
Large object at address: 0x77aee000 of length 36864 bytes
Free block at address: 0x77af7000 of length 8192 bytes
Large object at address: 0x77af9000 of length 143360 bytes
Large object at address: 0x77b1c000 of length 20480 bytes
Large object at address: 0x77b21000 of length 20480 bytes
Large object at address: 0x77b26000 of length 20480 bytes
Large object at address: 0x77b2b000 of length 20480 bytes
Large object at address: 0x77b30000 of length 20480 bytes
Large object at address: 0x77b35000 of length 20480 bytes
Large object at address: 0x77b3a000 of length 20480 bytes
Large object at address: 0x77b3f000 of length 20480 bytes
Large object at address: 0x77b44000 of length 20480 bytes
Large object at address: 0x77b49000 of length 20480 bytes
Large object at address: 0x77b4e000 of length 69632 bytes
Large object at address: 0x77b5f000 of length 36864 bytes
Large object at address: 0x77b68000 of length 69632 bytes
Large object at address: 0x77b79000 of length 36864 bytes
Large object at address: 0x77b82000 of length 69632 bytes
Free block at address: 0x77b93000 of length 12288 bytes
Large object at address: 0x77b96000 of length 20480 bytes
Large object at address: 0x77b9b000 of length 69632 bytes
Large object at address: 0x77bac000 of length 36864 bytes
Large object at address: 0x77bb5000 of length 69632 bytes
Large object at address: 0x77bc6000 of length 36864 bytes
Large object at address: 0x77bcf000 of length 69632 bytes
Large object at address: 0x77be0000 of length 36864 bytes
Large object at address: 0x77be9000 of length 20480 bytes
Large object at address: 0x77bee000 of length 20480 bytes
Large object at address: 0x77bf3000 of length 20480 bytes
Large object at address: 0x77bf8000 of length 20480 bytes
Large object at address: 0x77bfd000 of length 69632 bytes
Large object at address: 0x77c0e000 of length 36864 bytes
Large object at address: 0x77c17000 of length 69632 bytes
Large object at address: 0x77c28000 of length 36864 bytes
Large object at address: 0x77c31000 of length 69632 bytes
Large object at address: 0x77c42000 of length 36864 bytes
Large object at address: 0x77c4b000 of length 69632 bytes
Large object at address: 0x77c5c000 of length 36864 bytes
Large object at address: 0x77c65000 of length 36864 bytes
Free block at address: 0x77c6e000 of length 16384 bytes
Large object at address: 0x77c72000 of length 20480 bytes
Free block at address: 0x77c77000 of length 524730368 bytes

MemMap:
[MemMap: 0x12c00000+0x20000P prot=0x3 main space (region space)]
[MemMap: 0x70703000+0x13feP prot=0x3 /data/misc/apexdata/com.android.art/dalvik-cache/boot.art]
[MemMap: 0x71b01000+0xf49P prot=0x0 Boot image reservation]
[MemMap: 0x71b01000+0xf47P(4) prot=0x0 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat]
[MemMap: 0x72973000+0xd5P prot=0x3 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.vdex]
[MemMap: 0x72a48000+0x2P(2) prot=0x0 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat]
[MemMap: 0x72a4a000+0x660P prot=0x3 /data/misc/apexdata/com.android.art/dalvik-cache/boot-framework-adservices.art]
[MemMap: 0x730aa000+0x39P prot=0x0 Boot image reservation]
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A [MemMap: 0x730aa000+0x37P(2) prot=0x0 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework-adservices.oat]
[MemMap: 0x730bc000+0x25P prot=0x3 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework-adservices.vdex]
[MemMap: 0x730e1000+0x2P(2) prot=0x0 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework-adservices.oat]
[MemMap: 0x730e3000+0x61fP prot=0x3 zygote space]
[MemMap: 0x73702000+0x39e1P prot=0x3 non moving space]
[MemMap: 0x770e3000+0x20000P prot=0x3 free list large object space]
[MemMap: 0x970e3000+0x2000P prot=0x1 zygote-data-code-cache]
[MemMap: 0x990e3000+0x2000P prot=0x5 zygote-jit-code-cache]
[MemMap: 0x9b0e3000+0x2000P prot=0x1 data-code-cache]
[MemMap: 0x9d0e3000+0x2000P prot=0x5 jit-code-cache]
[MemMap: 0x9f0e3000+0x1bdP prot=0x3 /data/app/~~EkgDN7qzK9d2R3Xi-MHc6g==/com.embee.uk-DKkN2lE_hTB8UCAkT3kc7w==/oat/arm64/base.art]
[MemMap: 0x9f2a0000+0x1P prot=0x3 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/system@framework@com.android.location.provider.jar@classes.art]
[MemMap: 0xebad6000+0x1P prot=0x0 Sentinel fault page]
[MemMap: 0x721e894000+0x649P prot=0x1 classes9.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes9.dex]
[MemMap: 0x721eedd000+0x842P prot=0x1 classes8.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes8.dex]
[MemMap: 0x721f71f000+0x815P prot=0x1 classes7.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes7.dex]
[MemMap: 0x721ff34000+0x872P prot=0x1 classes6.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes6.dex]
[MemMap: 0x72207a6000+0x83fP prot=0x1 classes5.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes5.dex]
[MemMap: 0x7220fe5000+0x899P prot=0x1 classes4.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes4.dex]
[MemMap: 0x722187e000+0x837P prot=0x1 classes3.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes3.dex]
[MemMap: 0x72220b5000+0x81dP prot=0x1 classes2.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes2.dex]
[MemMap: 0x7223b06000+0x1d6P prot=0x1 classes10.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes10.dex]
[MemMap: 0x7223cdc000+0x76aP prot=0x1 classes.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk]
[MemMap: 0x7224446000+0x7aaP(5) prot=0x0 /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/oat/arm64/base.odex]
[MemMap: 0x7224949000+0x2a7P prot=0x3 /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/oat/arm64/base.vdex]
[MemMap: 0x7224bf0000+0x2P(2) prot=0x0 /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/oat/arm64/base.odex]
[MemMap: 0x7225825000+0xdP(5) prot=0x0 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/system@framework@com.android.location.provider.jar@classes.odex]
[MemMap: 0x7225831000+0x1P prot=0x3 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/system@framework@com.android.location.provider.jar@classes.vdex]
[MemMap: 0x7225832000+0x2P(2) prot=0x0 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/system@framework@com.android.location.provider.jar@classes.odex]
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A [MemMap: 0x726ee33000+0xd82P(5) prot=0x0 /data/app/~~EkgDN7qzK9d2R3Xi-MHc6g==/com.embee.uk-DKkN2lE_hTB8UCAkT3kc7w==/oat/arm64/base.odex]
[MemMap: 0x726f2a3000+0x912P prot=0x3 /data/app/~~EkgDN7qzK9d2R3Xi-MHc6g==/com.embee.uk-DKkN2lE_hTB8UCAkT3kc7w==/oat/arm64/base.vdex]
[MemMap: 0x726fbb5000+0x2P(2) prot=0x0 /data/app/~~EkgDN7qzK9d2R3Xi-MHc6g==/com.embee.uk-DKkN2lE_hTB8UCAkT3kc7w==/oat/arm64/base.odex]
[MemMap: 0x72c9bd8000+0x78P prot=0x1 classes.dex mapped directly in memory from /system/framework/org.apache.http.legacy.jar]
[MemMap: 0x72c9c50000+0x67P(5) prot=0x0 /data/user_de/0/com.google.android.gms/app_chimera/m/000003fa/oat/arm64/DynamiteLoader.odex]
[MemMap: 0x72c9cb5000+0x2P prot=0x3 /data/user_de/0/com.google.android.gms/app_chimera/m/000003fa/oat/arm64/DynamiteLoader.vdex]
[MemMap: 0x72c9cb7000+0x2P(2) prot=0x0 /data/user_de/0/com.google.android.gms/app_chimera/m/000003fa/oat/arm64/DynamiteLoader.odex]
[MemMap: 0x72cef18000+0xe8P prot=0x3 allocspace non moving space mark-bitmap 1]
[MemMap: 0x72cf000000+0x2000P prot=0x3 data-code-cache-rw]
[MemMap: 0x72d1000000+0x2000P prot=0x3 jit-code-cache-rw]
[MemMap: 0x72d3918000+0xe8P prot=0x3 allocspace non moving space live-bitmap 1]
[MemMap: 0x72dcb02000+0x20P~0xaP+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x72dffc0000+0x20P~0x18fP+0x20P~0xf5P+0x20P~0x29P+0x20P~0xb7P+0x20P~0x74P+0x20P prot=0x3 CompilerMetadata]
[MemMap: 0x72e0458000+0x100P prot=0x3 non-moving-space inter region ref bitmap]
[MemMap: 0x72e0558000+0x1000P prot=0x3 region-space inter region ref bitmap]
[MemMap: 0x72e1558000+0x200P prot=0x3 rb copying gc mark stack]
[MemMap: 0x72e1758000+0x800P prot=0x3 concurrent copying gc mark stack]
[MemMap: 0x72e1f58000+0x200P prot=0x3 rb copying gc mark stack]
[MemMap: 0x72e2158000+0x800P prot=0x3 concurrent copying gc mark stack]
[MemMap: 0x72e2958000+0x801P prot=0x3 live stack]
[MemMap: 0x72e3159000+0x801P prot=0x3 allocation stack]
[MemMap: 0x72e395a000+0x401P prot=0x3 card table]
[MemMap: 0x72e3d5b000+0x100P prot=0x3 large object free list space allocation info map]
[MemMap: 0x72e3e5b000+0x1000P prot=0x3 region space live bitmap]
[MemMap: 0x72e4e5b000+0x100P prot=0x3 allocspace zygote / non moving space mark-bitmap 0]
[MemMap: 0x72e4f5b000+0x100P prot=0x3 allocspace zygote / non moving space live-bitmap 0]
[MemMap: 0x72e505b000+0x11dP prot=0x1 classes.dex mapped directly in memory from /apex/com.android.wifi/javalib/framework-wifi.jar]
[MemMap: 0x72e5178000+0x194P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.tethering/javalib/framework-connectivity.jar]
[MemMap: 0x72e530c000+0xbcP prot=0x1 classes.dex mapped directly in memory from /apex/com.android.media/javalib/updatable-media.jar]
[MemMap: 0x72e53c8000+0xaeP prot=0x1 classes.dex mapped directly in memory from /apex/com.android.ipsec/javalib/android.net.ipsec.ike.jar]
[MemMap: 0x72e5476000+0x10fP prot=0x1 classes.dex mapped directly in memory from /apex/com.android.btservices/javalib/framework-bluetooth.jar]
[MemMap: 0x72e5585000+0x2cfP prot=0x1 classes.dex mapped directly in memory from /apex/com.android.i18n/javalib/core-icu4j.jar]
[MemMap: 0x72e5854000+0x93P prot=0x1 classes.dex mapped directly in memory from /system/framework/ims-common.jar]
[MemMap: 0x72e58e7000+0xc9P prot=0x1 classes.dex mapped directly in memory from /system/framework/voip-common.jar]
[MemMap: 0x72e59b0000+0x3b8P prot=0x1 classes.dex mapped directly in memory from /system/framework/telephony-common.jar]
[MemMap: 0x72e5d68000+0x14eP prot=0x1 classes5.dex mapped directly in memory from /system/framework/framework.jar!classes5.dex]
[MemMap: 0x72e5eb6000+0x965P prot=0x1 classes4.dex mapped directly in memory from /system/framework/framework.jar!classes4.dex]
[MemMap: 0x72e681b000+0x9cdP prot=0x1 classes3.dex mapped directly in memory from /system/framework/framework.jar!classes3.dex]
[MemMap: 0x72e71e8000+0x88aP prot=0x1 classes2.dex mapped directly in memory from /system/framework/framework.jar!classes2.dex]
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A [MemMap: 0x72e7a72000+0x917P prot=0x1 classes.dex mapped directly in memory from /system/framework/framework.jar]
[MemMap: 0x72e8389000+0x124P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.art/javalib/apache-xml.jar]
[MemMap: 0x72e84ad000+0x553P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.art/javalib/core-oj.jar]
[MemMap: 0x72e9696000+0x16aP prot=0x1 classes.dex mapped directly in memory from /apex/com.android.art/javalib/bouncycastle.jar]
[MemMap: 0x757de39000+0x80P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.healthfitness/javalib/framework-healthfitness.jar]
[MemMap: 0x757deb9000+0x8cP prot=0x1 classes.dex mapped directly in memory from /apex/com.android.art/javalib/core-libart.jar]
[MemMap: 0x757df61000+0x20P~0x5fP+0x20P~0x321cP+0x40P(2) prot=0x3 CompilerMetadata]
[MemMap: 0x7581307000+0x20P~0x48P+0x20P~0x5eP+0x20P~0xd3P+0x40P(2)~0xd8P+0x20P~0x89P+0x40P(2) prot=0x3 LinearAlloc]
[MemMap: 0x7581922000+0x60P(3)~0x16P+0x40P(2)~0x1028P+0x40P(2)~0x35P+0x20P~0x14bP+0x20P~0x341dP+0x20P~0x1f6P+0x20P~0x10P+0x20P~0x17dP+0x20P prot=0x3 CompilerMetadata]
[MemMap: 0x75864fd000+0x20P~0x35a6P+0x40P(2)~0x4dP+0x20P~0x1b8bP+0x20P~0x299P+0x20P~0x243P+0x40P(2) prot=0x3 LinearAlloc]
[MemMap: 0x758bd5e000+0xdP prot=0x1 classes11.dex mapped directly in memory from /data/app/~~L5WQrDr3Xor4MBTM-bNjAA==/com.google.android.gms--Mz6U-sq_ufcaC6_L06Kdw==/base.apk!classes11.dex]
[MemMap: 0x758bd6b000+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x758bda1000+0x40P(2)~0x3020P+0x40P(2)~0xf6P+0x20P~0x62P+0x20P~0xd9P+0x40P(2)~0x28P+0x20P~0xaP+0x20P prot=0x3 CompilerMetadata]
[MemMap: 0x7590a60000+0x20P~0x44P+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x7590ae9000+0x4dP prot=0x1 classes.dex mapped directly in memory from /data/user_de/0/com.google.android.gms/app_chimera/m/000003ff/MeasurementDynamite.apk]
[MemMap: 0x7590be5000+0x20P~0x35eP+0x20P~0x95P+0x20P~0x7fP+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x7591220000+0x25P prot=0x1 classes.dex mapped directly in memory from /data/user_de/0/com.google.android.gms/app_chimera/m/000003fa/DynamiteLoader.apk]
[MemMap: 0x75912aa000+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x7591970000+0x9P prot=0x1 classes2.dex mapped directly in memory from /data/user_de/0/com.google.android.gms/app_chimera/m/000003ff/MeasurementDynamite.apk!classes2.dex]
[MemMap: 0x75949b7000+0x7P prot=0x1 classes.dex mapped directly in memory from /system/framework/com.android.location.provider.jar]
[MemMap: 0x7594ae7000+0x20P~0x2bddP+0x20P~0xfeP+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x7599802000+0x3P prot=0x1 classes.dex mapped directly in memory from /system/framework/com.android.media.remotedisplay.jar]
[MemMap: 0x7599805000+0x64P prot=0x3 indirect ref table]
[MemMap: 0x759986d000+0x6P prot=0x1 DEX data]
[MemMap: 0x7599893000+0x20P~0x18P+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x75998f5000+0x1P prot=0x3 local ref table]
[MemMap: 0x759990b000+0x20P~0xf5P+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x7599be4000+0x3P prot=0x3 /system/framework/oat/arm64/org.apache.http.legacy.vdex]
[MemMap: 0x759c800000+0x1P prot=0x1 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/system@framework@com.android.location.provider.jar@classes.art]
[MemMap: 0x759c801000+0x1P prot=0x3 /system/framework/oat/arm64/com.android.media.remotedisplay.vdex]
[MemMap: 0x759c814000+0x64P prot=0x3 indirect ref table]
[MemMap: 0x759c87b000+0x3P prot=0x1 DEX data]
[MemMap: 0x759c93f000+0x20P~0x1eP+0x20P prot=0x3 CompilerMetadata]
[MemMap: 0x759c9a0000+0x2P(2)~0x5eP+0x1P~0x3dcP+0x2P(2) prot=0x3 local ref table]
[MemMap: 0x759ee00000+0x4P prot=0x3 /data/user_de/0/com.google.android.gms/app_chimera/m/000003ff/oat/arm64/MeasurementDynamite.vdex]
[MemMap: 0x759f02a000+0x1P prot=0x3 local ref table]
[MemMap: 0x759f080000+0x4P prot=0x1 /data/app/~~EkgDN7qzK9d2R3Xi-MHc6g==/com.embee.uk-DKkN2lE_hTB8UCAkT3kc7w==/oat/arm64/base.art]
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A [MemMap: 0x759f169000+0x63P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.tethering/javalib/framework-connectivity-t.jar]
[MemMap: 0x759f1d0000+0x1P prot=0x3 local ref table]
[MemMap: 0x759f1e0000+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x75a2606000+0x9P prot=0x1 classes.dex mapped directly in memory from /system/framework/android.test.base.jar]
[MemMap: 0x75a260f000+0x8P prot=0x1 classes.dex mapped directly in memory from /system/framework/android.hidl.manager-V1.0-java.jar]
[MemMap: 0x75a2b8e000+0x72P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.conscrypt/javalib/conscrypt.jar]
[MemMap: 0x75a4f17000+0x4P prot=0x1 classes.dex mapped directly in memory from /system/framework/android.hidl.base-V1.0-java.jar]
[MemMap: 0x75a4f23000+0x20P~0x15P+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x75a4fd5000+0x1P prot=0x3 mod union bitmap]
[MemMap: 0x75a4fea000+0x20P prot=0x3 LinearAlloc]
[MemMap: 0x75a500a000+0x10P(4)~0x4P+0x24P(9)~0xeP+0x3cP(15)~0x34P+0xcP(3)~0x14P+0x34P(13)~0x2aP+0x24P(9)~0x1eP+0x34P(13)~0xdP+0x18P(6)~0x7P+0x38P(14)~0x11P+0x38P(14) prot=0x3 thread local mark stack]
[MemMap: 0x75a527c000+0x28P(10)~0x17P+0x2cP(11)~0xcP+0x14P(5)~0xdP+0x54P(21)~0x18P+0xcP(3)~0xfP+0x54P(21)~0x9P+0x8P(2)~0x9bP+0x2cP(11)~0xcP+0x4cP(19)~0x7P+0x28P(10) prot=0x3 thread local mark stack]
[MemMap: 0x75a557f000+0x30P(12)~0x12P+0x8P(2)~0x1cP+0x30P(12)~0x11P+0x48P(18)~0x190P+0x14P(5)~0x7P+0x48P(18)~0x10P+0x30P(12)~0x5P+0x50P(20) prot=0x3 thread local mark stack]
[MemMap: 0x75a58fb000+0x1P prot=0x3 /system/framework/oat/arm64/android.test.base.vdex]
[MemMap: 0x75a58fc000+0x24P(9) prot=0x3 thread local mark stack]
[MemMap: 0x75a5924000+0x1P prot=0x3 /system/framework/oat/arm64/android.hidl.manager-V1.0-java.vdex]
[MemMap: 0x75a5926000+0x20P(8) prot=0x3 thread local mark stack]
[MemMap: 0x75a5961000+0x1P prot=0x3 /system/framework/oat/arm64/android.hidl.base-V1.0-java.vdex]
[MemMap: 0x75a5962000+0x34P(13)~0xaP+0x38P(14)~0x6P+0x28P(10)~0x1cP+0x44P(17)~0x8P+0x28P(10)~0x17P+0x24P(9)~0x1fP+0x18P(6)~0x2fP+0x24P(9)~0x4P+0x3cP(15)~0x1aP+0x34P(13) prot=0x3 thread local mark stack]
[MemMap: 0x75a5bed000+0xcP(3) prot=0x3 thread local mark stack]
[MemMap: 0x75a5bf9000+0x40P prot=0x3 mark stack]
[MemMap: 0x75a5c7a000+0x18P(6) prot=0x3 thread local mark stack]
[MemMap: 0x75a5c92000+0x63P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.adservices/javalib/framework-adservices.jar]
[MemMap: 0x75a5d04000+0x3cP(15)~0xb7P+0x28P(10)~0x2P+0x20P(8) prot=0x3 thread local mark stack]
[MemMap: 0x75a5e41000+0x20P prot=0x3 large marked objects]
[MemMap: 0x75a5e61000+0x20P prot=0x3 large live objects]
[MemMap: 0x75a5ec4000+0x8P(2) prot=0x3 thread local mark stack]
[MemMap: 0x75a5f0f000+0x14P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.virt/javalib/framework-virtualization.jar]
[MemMap: 0x75a5f27000+0x64P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.art/javalib/okhttp.jar]
[MemMap: 0x75a5f94000+0xcP(3) prot=0x3 thread local mark stack]
[MemMap: 0x75a5fa0000+0x22P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.uwb/javalib/framework-uwb.jar]
[MemMap: 0x75a5fdb000+0x4P prot=0x3 thread local mark stack]
[MemMap: 0x75a5fdf000+0x35P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.permission/javalib/framework-permission-s.jar]
[MemMap: 0x75a603a000+0x10P(4)~0x32P+0x10P(4) prot=0x3 thread local mark stack]
[MemMap: 0x75a608c000+0x2dP prot=0x1 classes.dex mapped directly in memory from /apex/com.android.ondevicepersonalization/javalib/framework-ondevicepersonalization.jar]
[MemMap: 0x75a60bc000+0x4P prot=0x3 thread local mark stack]
[MemMap: 0x75a60c2000+0x19P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.os.statsd/javalib/framework-statsd.jar]
[MemMap: 0x75a60db000+0x31P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.appsearch/javalib/framework-appsearch.jar]
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A [MemMap: 0x75a616a000+0x23P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.configinfrastructure/javalib/framework-configinfrastructure.jar]
[MemMap: 0x75a618e000+0x10P(4)~0x4P+0x4P prot=0x3 thread local mark stack]
[MemMap: 0x75a61a6000+0x14P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.mediaprovider/javalib/framework-mediaprovider.jar]
[MemMap: 0x75a61ba000+0x48P prot=0x1 classes.dex mapped directly in memory from /system/framework/ext.jar]
[MemMap: 0x75a6208000+0x11P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.tethering/javalib/framework-tethering.jar]
[MemMap: 0x75a6219000+0x12P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.adservices/javalib/framework-sdksandbox.jar]
[MemMap: 0x75a622b000+0x23P prot=0x1 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.art]
[MemMap: 0x75a62cc000+0x8P(2)~0x8aP+0x4P prot=0x3 thread local mark stack]
[MemMap: 0x75a6362000+0xcP prot=0x1 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework-adservices.art]
[MemMap: 0x75a6372000+0x1P prot=0x3 local ref table]
[MemMap: 0x75a6410000+0x2P prot=0x3 concurrent copying sweep array free buffer]
[MemMap: 0x75a6412000+0x7P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.devicelock/javalib/framework-devicelock.jar]
[MemMap: 0x75a6419000+0x1P~0x1P+0x1P~0x1P+0x2P(2) prot=0x3 local ref table]
[MemMap: 0x75a65fa000+0x4P~0x130P+0x4P prot=0x3 thread local mark stack]
[MemMap: 0x75a6737000+0x2P prot=0x3 concurrent copying sweep array free buffer]
[MemMap: 0x75a6739000+0x4P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.scheduling/javalib/framework-scheduling.jar]
[MemMap: 0x75a673f000+0x1P prot=0x3 local ref table]
[MemMap: 0x75a6740000+0x2P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.sdkext/javalib/framework-sdkextensions.jar]
[MemMap: 0x75a6742000+0x1P prot=0x1 classes.dex mapped directly in memory from /apex/com.android.permission/javalib/framework-permission.jar]
[MemMap: 0x75a78f7000+0x1P prot=0x1 classes.dex mapped directly in memory from /system/framework/framework-graphics.jar]
[MemMap: 0x75a78f8000+0x1P prot=0x3 local ref table]'
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A x0 0000000000000000 x1 00000000000064c9 x2 0000000000000006 x3 00000072cac11020
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A x4 00000000676f6c62 x5 00000000676f6c62 x6 00000000676f6c62 x7 b4000073bd214516
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A x8 00000000000000f0 x9 00000075a2674050 x10 0000000000000001 x11 00000075a26bdb60
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A x12 000000000000937a x13 000000000000074f x14 00000072cac0fe30 x15 0000074608e2270a
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A x16 00000075a2729cf8 x17 00000075a2706470 x18 0000007278160000 x19 00000000000064bd
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A x20 00000000000064c9 x21 00000000ffffffff x22 000000000000000b x23 0000000000000001
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A x24 00000072e8abfab4 x25 0000000000000002 x26 0000000000000001 x27 000000000000000b
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A x28 00000072e9616000 x29 00000072cac110a0
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A lr 00000075a26ae178 sp 00000072cac11000 pc 00000075a26ae1a4 pst 0000000000001000
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A 23 total frames
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A backtrace:
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A #00 pc 000000000005c1a4 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A #01 pc 00000000007845a0 /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+904) (BuildId: a5fcf27f4a71b07dff05c648ad58e3cd)
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A #02 pc 00000000000357d0 /apex/com.google.mainline.primary.libs@340941000/lib64/

libbase.so/75d3253827fcfd7a8d7b02ad45991611ec4ca424c0278e13e8acfad4d14e597a3ecff6c0caa2b785c73838528ee6e9c2b313240ff895f50ee39b1d7bc10f390a/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+80) (BuildId: 6f67f69ff36b970d0b831cfdab3b578d)
2023-10-16 17:46:08.533 28944-28944 DEBUG pid-28944 A #03 pc 0000000000034d58 /apex/com.google.mainline.primary.libs@340941000/lib64/

lo...@google.com <lo...@google.com> #31Oct 16, 2023 11:47PM

Please note that the crash reported in comment #21 and comment #30 are different from the originally reported issue. They all crash in the GC so look similar but have different symptoms and therefore should be tackled as such.

The most important thing to fix such issues is if it's reproducible. If you are able to then please provide us steps to do so. Also, please tell us if sept 9 launch of your app was the first launch or was it an update of the app. If it was an app update, then please closely look into the changes in the app.

lo...@google.com <lo...@google.com> #32Oct 17, 2023 12:50AM

How easy is it that for you to reproduce comment #23? You should definitely pursue this with flutter team.

qi...@trip.com <qi...@trip.com> #33Oct 17, 2023 12:55AM

Hi Google Team,

Let's back to the issue I'v reported to you originally, there are two kind of crashes we encountered, one is the "signal 11 (SIGSEGV), code 1 (SEGV_MAPERR)", please see the tombstones(tombstone_06, tombstone_09) I'v provided in the comments above. the other one is "signal 6 (SIGABRT), code -1 (SI_QUEUE)", please see the tombstones(tombstone_08).

We reproduced this crash via kind of monkey test, so can not provide the steps to reproduce(it is also because we dont know which part of the code operate the mem addr in a wrong way). So maybe you can reproduce it in a similar way(monkey test). Please be noticed, as mentioned above, This problem suppose to be only occurred on the devices with google play store available according to the distribution of impact users(This is the originally reason I suspect is related to the ART update, plus the 21st floor also says it was affected by a similar problem on the same day).

btw, the logs I'v provided in #1 and #10 are similar to #30.

If you need any more information about solving this problem, please don't hesitate to ask me!

Looking forward to your replies.

Thanks.

qi...@trip.com <qi...@trip.com> #34Oct 17, 2023 01:11AM

Hi google team,

Only reproduce once for

comment #23

, and for #23 is another app of our company, the latest version of this app was released on 29th Mar, also affected by this issue suddenly on 12th Sep, which means no new version release, and for my App, please see

comment #15

.

So I'm sure that no new release from my side leads to this issue. And similarweb team also affected, as I mentioned in

comment #22

(same day affected, similar issue logs). I suppose ART new release or OTAs system update leads to this issue.

ng...@google.com <ng...@google.com> #35Oct 17, 2023 08:24AM

Yes, the ART update in August on Android12+ devices likely triggered the issues, though the bug may have been latent and it is now exposing it. It'd be great if we could repro the issue.

de...@similarweb.com <de...@similarweb.com> #36Oct 17, 2023 08:27AM

Although I don't know the nuances of the art garbage collection and can't say if the underlying issue is different or not, but the broader issue appears to be various native crashes (variation of SIGSEGV, SIGABRT and SIGBUS). Similar "September 12 native crashes" issues were reported in

https://issuetracker.google.com/issues/300840851. In that ticket the issue was with react-native apps, but what I believe it comes down to is some security patches on the devices arriving in Aug/Sep 2023 which caused issues in some libraries used by various apps.

https://issuetracker.google.com/issues/300840851 specifically mentions issues occurring on older app versions (pre-Sep 12) and also reverting to earlier app versions (thinking the issue was introduced by app changes) that did not help.
One library that appears to be causing issues is Sentry

https://status.sentry.io/incidents/912c0wsfxtgq, although our app is not using, so I guess there could other libraries having the same kinds of issues triggered by August ART update.

lo...@google.com <lo...@google.com> #37Oct 17, 2023 11:36PM

A little background: All the crashes pointed out in comment #33, or the fact that they are SIGSEGV/SIGABRT/SIGBUS, they all point to a memory-corruption type issue. We are very much interested in fixing such issue, if it is arising out of something in ART. However, in the past there have been several times when such issues happened due to bugs in the app. Also, the GC has been in use since Android 8. I'm not suggesting that it can't be an issue in the GC (or in ART), but it not very likely.

In this case, since the crashes increased steadily immediately after app update on Sept9, it elevates our hope that maybe it is easier to pin-point the issue to some change in update. Otherwise, given the steady rise in crashes, there must be a way to reproduce this locally, so that I can add logs to the code and reproduce this locally for fix the issue.

BTW, the crashes reported in https://issuetracker.google.com/issues/300840851 are unrelated to the GC. So they should not be merged with the crashes reported in this bug.

Again, please if possible try the pre-sep12 app version and see if the GC crashes are still happening. And if they are happening, then please help us with some steps to reproduce. We are fine if it takes even 50 retries to hit it, as long as there are meaningful steps of triggering the crash.

Also, please get in touch with flutter team about the MTE crash of comment #23

qi...@trip.com <qi...@trip.com> #38Oct 18, 2023 12:46AM

Hi google Team,

Thank you for the relay.

As I’v concluded our issue situations in

comment #33

and above, all online versions are affected, not only for the version we released on 7th Sep, and all the apps published in Google play store of our company are affected too and the increase trend are similar among all versions. Which means our users no matter what version they are using are affected after they updated ART(we suppose it is related as we discussed above, please see

comment #34

), it’s likely that some problem might be hidden in the old version of ART, but now is exposing.

For reproducing, this issue occurred very randomly, and we can only reproduce via kind of monkey test. We are trying to provide our test tool for you to reproduce.

qi...@trip.com <qi...@trip.com> #39Oct 18, 2023 12:54PM

Hi Google Team,

Can you provide the code diff of ART version upgrade in Google source?

Thanks

qi...@trip.com <qi...@trip.com> #40
Restricted
Oct 18, 2023 01:22PM

Comment has been deleted.

deleted

Restricted

0 B

qi...@trip.com <qi...@trip.com> #41Oct 19, 2023 07:43AM

Hi Google Team,

If you have any trouble with following steps of how to reproduce in

comment #40

, please DO NOT hesitate to contact me.

Thanks.

lo...@google.com <lo...@google.com> #42Oct 19, 2023 11:39PM

Hi, I tried the script that you provided in comment #40, but faced issues with it. It results in following error:

start explore
Options：-p ctrip.english --running-minutes 30 --support-crn --pre-input --module_list ibu-android-hotel 
processOptions：-p
processOptions：--running-minutes
processOptions：--support-crn
processOptions：--pre-input
processOptions：--module_list
[2023-10-19 12:38:02,227][INFO][__main__:157] [iaet] adb -s 21081FDH3001QH pull /sdcard/iAET-finish-tag.log result-21081FDH3001QH/iaet-finish-tag.log
adb: error: failed to stat remote object '/sdcard/iAET-finish-tag.log': No such file or directory
[2023-10-19 12:38:02,243][INFO][__main__:420] left running time 30 min
[2023-10-19 12:38:02,244][INFO][__main__:423] explore cmd: adb -s 21081FDH3001QH shell CLASSPATH='/data/local/tmp/iAET.jar' '/system/bin/app_process' '/data/local/tmp/iAET.jar' com.ibu.testing.Main -p ctrip.english --running-minutes 30 --support-crn --pre-input --module_list ibu-android-hotel

qi...@trip.com <qi...@trip.com> #43Oct 20, 2023 02:25AM

Hi Google Team,

What is the phenomenon after you got this log? I think it would be started launch the app, and accessed page randomly, and would be continuous explored for 30 minutes.

DO NOT worry about "adb: error: failed to stat remote object '/sdcard/iAET-finish-tag.log': No such file or directory", which will not interrupt the auto explore testing.

mu...@gmail.com <mu...@gmail.com> #44Oct 20, 2023 07:58AM

Hi Google team,

Our app is also facing the exact same issue that was discussed by @

trip.com, we also saw a huge spike in native crashes of 2 kinds on Sep 8, 2023, and currently. It shows the app's bad behavior. This crash is reported in the Google Play console, not in Firebase and unable to reproduce at my end. Looking forward to your reply

Carsh 1:

backtrace:
#00 pc 0x000000000024b03c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1720)
#01 pc 0x0000000000247c7c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#02 pc 0x0000000000244640 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#03 pc 0x00000000003bde5c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#04 pc 0x00000000003bb8c0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#05 pc 0x00000000003bb114 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#06 pc 0x00000000003baf98 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#07 pc 0x0000000000393830 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#08 pc 0x000000000037b514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#09 pc 0x00000000006d8768 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#10 pc 0x00000000006ab49c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#11 pc 0x00000000004ee2f8 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#12 pc 0x00000000003605a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#13 pc 0x000000000034b930 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#14 pc 0x00000000004f3e38 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#15 pc 0x00000000000c1b40 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+224)
#16 pc 0x0000000000054f20 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

Crash 2

backtrace:
#00 pc 0x000000000007caa4 /apex/com.android.runtime/lib64/bionic/libc.so (abort+180)
#01 pc 0x00000000007910cc /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+1380)
#02 pc 0x00000000000357d0 /apex/com.google.mainline.primary.libs@340817060/lib64/

libbase.so/5cbcb48dc4a1ceb71422e6e07e9f12bb952b969b1d4789234c07c3b6241482a6f3a09733800aa23496f3dcd787dd1ded12735fe6dc5b3049d050d96b769bc049/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+80)
#03 pc 0x0000000000034d58 /apex/com.google.mainline.primary.libs@340817060/lib64/

libbase.so/5cbcb48dc4a1ceb71422e6e07e9f12bb952b969b1d4789234c07c3b6241482a6f3a09733800aa23496f3dcd787dd1ded12735fe6dc5b3049d050d96b769bc049/libbase.so (android::base::LogMessage::~LogMessage()+352)
#04 pc 0x0000000000207724 /apex/com.android.art/lib64/libart.so (art::gc::Verification::LogHeapCorruption(art::ObjPtr<art::mirror::Object>, art::MemberOffset, art::mirror::Object*, bool) const+1976)
#05 pc 0x000000000024b740 /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&)+1452)
#06 pc 0x000000000024b0a0 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1820)
#07 pc 0x0000000000247c7c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#08 pc 0x0000000000244640 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#09 pc 0x00000000003bde5c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#10 pc 0x00000000003bb8c0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#11 pc 0x00000000003bb114 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#12 pc 0x00000000003baf98 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#13 pc 0x0000000000393830 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#14 pc 0x000000000041d514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#15 pc 0x00000000006f9c18 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#16 pc 0x00000000006ccb4c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#17 pc 0x000000000050f628 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#18 pc 0x00000000003605a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#19 pc 0x000000000034b930 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#20 pc 0x00000000004f3e38 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#21 pc 0x00000000000eb720 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#22 pc 0x000000000007e2d0 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

deleted

Restricted

0 B

mu...@gmail.com <mu...@gmail.com> #45Oct 20, 2023 07:59AM

Comment has been deleted.

Message last modified on Oct 20, 2023 08:00AM

mu...@gmail.com <mu...@gmail.com> #46Oct 20, 2023 08:00AM

The crash is only above of android versions 13 and 13+, below 13 is working fine as of now.

lo...@google.com <lo...@google.com> #47Oct 20, 2023 06:05PM

Hi,

Please get us the following details:

What device are you seeing the crashes. If a lot of them then, please let us know which Pixel phones it is happening on?
When you say below 13 it doesn't happen, can you confirm if the ART module is updated or not. Or, which date it was updated on? Are you sure that android 12 device with the sept8 module update is also not causing the crash?
If you are seeing so many crashes, please provide some steps to reproduce

lo...@google.com <lo...@google.com> #48Oct 20, 2023 06:06PM

Also, what is the app's name?

lo...@google.com <lo...@google.com> #49Oct 20, 2023 06:07PM

Regarding comment #43, I started getting this log immediately after the app was downloaded and installed by the script. And as you said, it continued with this log message for the entire 30mins.

mu...@gmail.com <mu...@gmail.com> #50Oct 21, 2023 07:33AM

Regarding #44 query and #47 queries findings
The app's name is Jeevansathi

#1. Most crashing devices include OnePlus, Oppo, Vivo, Samsung, and more, but notably, I couldn't find any instances of Pixel phones on that list of problematic devices.

#2. We, as developers, haven't introduced any changes related to ART (Android Runtime) and are unsure about how these issues can be resolved. We apologize for any inconvenience. Sharing data for Android versions 12 and 13, it's evident that no users were affected on Android 12, but approximately 48,000 users encountered issues on Android 13.

#3. I've observed numerous crashes on the Play Console but not on the Firebase Crashlytics Center. Furthermore, I haven't been able to reproduce these issues yet. We are diligently attempting to reproduce them, and if successful, we will certainly provide steps to replicate them. We apologize for any inconvenience.

Our application incorporates several third-party SDKs that utilize native libraries like Spectrum, Flutter, and Agora. It's possible that these are contributing to the problem, although we haven't made any recent updates to these components. Curiously, these issues seem to be isolated to Android 13, and we have been unable to reproduce them thus far.

I also see ART improvement updates here

https://source.android.com/docs/core/runtime#Improved_GC. Updates related to GC improvement and the crash are also related to GC so maybe the cause is an ART update please help to find it.

I am attaching again the Crash details here

Crash Details:

[libart. so] art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()
SIGSEGV
In production
This issue is affecting users in a production track and should be prioritized
2 versions currently available for download
Production
461 (38.7.8)
460 (38.7.7)
Open testing
461 (38.7.8)
460 (38.7.7)
Internal testing
461 (38.7.8)
28 versions previously available for download
Other versions
459 (38.7.7)
458 (38.7.6)
457 (38.7.5)
456 (38.7.4)
454 (38.7.3)
23 others

Inner details:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 17541 >>>

com.jeevansathi.android <<<

backtrace:
#00 pc 0x000000000024b03c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1720)
#01 pc 0x0000000000247c7c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#02 pc 0x0000000000244640 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#03 pc 0x00000000003bde5c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#04 pc 0x00000000003bb8c0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#05 pc 0x00000000003bb114 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#06 pc 0x00000000003baf98 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#07 pc 0x0000000000393830 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#08 pc 0x000000000037b514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#09 pc 0x00000000006d8768 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#10 pc 0x00000000006ab49c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#11 pc 0x00000000004ee2f8 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#12 pc 0x00000000003605a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#13 pc 0x000000000034b930 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#14 pc 0x00000000004f3e38 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#15 pc 0x00000000000c1b40 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+224)
#16 pc 0x0000000000054f20 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

mu...@gmail.com <mu...@gmail.com> #51Oct 21, 2023 07:34AM

Please find the crash metrics data device-wise.

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

mm...@gmail.com <mm...@gmail.com> #52Oct 21, 2023 03:09PM

This is also happening for our app and its start almost on same dates 11th/12th Sep. Its random crash, nothing in firebase crashlytics. Happening for many users, there is no update for our app on those dates. Sometimes , updating Chrome and Android webview can solve the issue temporarily

vi...@gmail.com <vi...@gmail.com> #53Oct 23, 2023 08:13AM

Hey Google, Waiting for an update from your side.

qi...@trip.com <qi...@trip.com> #54Oct 23, 2023 09:08AM

Regarding

comment #42

and

comment #49

, please check as the following:
1. Please use a device with android 12 or above
2. Please keep the device unlocked when executing script
3. If the phenomenon you mentioned in

comment #49

still exist, please try to re-run the script, and change the "--running-minutes 30" to "--running-minutes 3" to reduce the executing time for more efficient retries. I reproduced once on a new device, but after I retried, it works.

If you have any troubles, please DO NOT hesitate to contact me.

Thanks.

qi...@trip.com <qi...@trip.com> #55Oct 23, 2023 10:22AM

Hi Google Team,

As we are continuously trying to reproduce this issue locally, we got something new, please help to check the backtrace in below and tombstone file whether it is useful to identify the cause.

"// Abort message: 'Check failed: region_space_alloc_size <= space::RegionSpace::kRegionSize (region_space_alloc_size=2147483648, space::RegionSpace::kRegionSize=262144) '"

// keyProcess: 0
// *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
// Build fingerprint: 'OPPO/PGT110/OP5605:14/UP1A.230620.001/T.13b789a_e_11:user/release-keys'
// Revision: '0'
// ABI: 'arm64'
// Timestamp: 2023-10-23 17:24:27.387663673+0800
// Process uptime: 51s
// Cmdline: ctrip.english
// pid: 29339, tid: 31441, name: HeapTaskDaemon >>> ctrip.english <<<
// uid: 10326
// tagged_addr_ctrl: 000000000007fff1 (PR_TAGGED_ADDR_ENABLE, mask 0xfffe)
// pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY)
// signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
// Abort message: 'Check failed: region_space_alloc_size <= space::RegionSpace::kRegionSize (region_space_alloc_size=2147483648, space::RegionSpace::kRegionSize=262144) '
// x0 0000000000000000 x1 0000000000007ad1 x2 0000000000000006 x3 0000007096d0b120
// x4 0000000000000001 x5 0000000000000001 x6 0000000000000001 x7 0a0000721c2e2156
// x8 00000000000000f0 x9 00000074474925b0 x10 0000000000000001 x11 00000074474ddb80
// x12 0000007447557490 x13 0000000000008af6 x14 0000000000000018 x15 0000000000000000
// x16 000000744754ad28 x17 0000007447526640 x18 0000007094668000 x19 00000000000000ac
// x20 00000000000000b2 x21 000000000000729b x22 0000000000007ad1 x23 00000000ffffffff
// x24 0000007105149aa3 x25 0000000000000017 x26 000000710513cdaa x27 0000007105a96000
// x28 0000007105133eda x29 0000007096d0b1a0
// lr 00000074474ce0d4 sp 0000007096d0b100 pc 00000074474ce104 pst 0000000000001000
// 24 total frames
// backtrace:
// #00 pc 000000000005e104 /apex/com.android.runtime/lib64/bionic/libc.so (abort+180) (BuildId: c39cb13d983c4e6b87d656f5d54b4b5a)
// #01 pc 0000000000651f10 /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+1536) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #02 pc 0000000000019910 /apex/com.android.art/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+80) (BuildId: 45adbd703b874320052f8d3c7560a2ad)
// #03 pc 0000000000018df4 /apex/com.android.art/lib64/libbase.so (android::base::LogMessage::~LogMessage()+356) (BuildId: 45adbd703b874320052f8d3c7560a2ad)
// #04 pc 0000000000321ce4 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::Copy(art::Thread*, art::mirror::Object*, art::mirror::Object*, art::MemberOffset)+5972) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #05 pc 0000000000328d18 /apex/com.android.art/lib64/libart.so (void art::gc::collector::ConcurrentCopying::Process<true>(art::mirror::Object*, art::MemberOffset)+552) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #06 pc 00000000003282a0 /apex/com.android.art/lib64/libart.so (void art::mirror::ObjectArray<art::mirror::Object>::VisitReferences<art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&)+64) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #07 pc 000000000031665c /apex/com.android.art/lib64/libart.so (void art::gc::accounting::SpaceBitmap<8ul>::VisitMarkedRange<false, art::gc::collector::ConcurrentCopying::CopyingPhase()::$_1&>(unsigned long, unsigned long, art::gc::collector::ConcurrentCopying::CopyingPhase()::$_1&) const (.__uniq.219178288367021339109957061695789229157)+332) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #08 pc 00000000003112b8 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::CopyingPhase()+1560) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #09 pc 000000000030e40c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+1116) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #10 pc 00000000003363c0 /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+304) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #11 pc 0000000000371454 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+2388) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #12 pc 0000000000382744 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+164) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #13 pc 00000000003893d8 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #14 pc 00000000003c4560 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+64) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #15 pc 0000000000011d14 /system/framework/arm64/boot-core-libart.oat (art_jni_trampoline+116) (BuildId: 709e31db4f966fa63389c817f319bd030c02f543)
// #16 pc 000000000004c1f8 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200) (BuildId: 709e31db4f966fa63389c817f319bd030c02f543)
// #17 pc 00000000000216fc /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$Daemon.run+172) (BuildId: 709e31db4f966fa63389c817f319bd030c02f543)
// #18 pc 0000000000178638 /system/framework/arm64/boot.oat (java.lang.Thread.run+72) (BuildId: bf71e19c0013c2aa9344374a9f667c45ad6ef509)
// #19 pc 00000000002109a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #20 pc 000000000025470c /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+172) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #21 pc 00000000006a1bb8 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1416) (BuildId: f86ed07b017bc3c31ab51f9794258865)
// #22 pc 00000000000cbde0 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208) (BuildId: c39cb13d983c4e6b87d656f5d54b4b5a)
// #23 pc 000000000005fa50 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: c39cb13d983c4e6b87d656f5d54b4b5a)
//

deleted

Restricted

0 B

lo...@google.com <lo...@google.com> #56Oct 23, 2023 11:23PM

Hi Mukesh (comment #50 and #51), Thanks for the data. This is helpful. It is very interesting that you are not seeing Android 12. Can you please get the following data:

Can you find data for Android 12 as well. The graphs you shared has for 12L and not 12
If the ART mainline module was updated on Android 12 devices as well, and if the issue is in the GC code, then Android 12 devices should also have seen crashes. Therefore, to figure out if the issue is in the GC code or elsewhere, it is important to know if the Android 12 devices has ART module updated or not, and if so then which version. Can you please try to get this info?

lo...@google.com <lo...@google.com> #57Oct 23, 2023 11:24PM

Re comment #55, thanks this can be helpful. I'll take a look.

In meantime, can you also confirm if the issues your are seeing are seen on Android 12 devices or not? This is a very important information. So please try to get this.

lo...@google.com <lo...@google.com> #58Oct 23, 2023 11:32PM

Please note that Active APEX packages: in the bugreport tells which ART module version is installed on the device. Getting this info would be very helpful.

qi...@trip.com <qi...@trip.com> #59Oct 24, 2023 07:06AM

Reply to

comment #57

.

As I originally reported to you, these issues appear on Android 12 and above(as the title of this issue has mentioned), and for additional of

comment #55

, we reproduced on a OPPO MTE device with android 14 installed.

In terms of ART version, you can get the build id in tombstone file, which is "BuildId: f86ed07b017bc3c31ab51f9794258865".

Thanks.

qi...@trip.com <qi...@trip.com> #60Oct 24, 2023 07:10AM

Hi Google Team,

Do you reproduce this issue properly via the script I'v provided in

comment #40

, regarding

comment #49

you encountered when you executed the script, please see

comment #54

.

Looking forward for your reply!

Thanks.

lo...@google.com <lo...@google.com> #61Oct 25, 2023 01:03AM

Since you have better control over the script, I'd suggest that you try to reproduce the issue at your end. I have been busy with trying to find the issue that is causing the crash and there are few things that I have managed to fix. These will be latest available in the January 2024 ART module. Unfortunately, given the general nature of these crashes, I can't be certain at this point that those fixes will be sufficient to fix the issues that you all are encountering. The best would have been to have steps to reliably reproduce the issue.

qi...@trip.com <qi...@trip.com> #62Oct 25, 2023 03:55AM

Hi Google Team,

It's wonderful that you've found some of the causes of these issues. We tried to use elimination to narrow it down but failed while feeling it maybe a multi-point that leads to these issues, I will continuously trying to reproduce and provide logs to you.

Thank you for have been investigating the causes. Looking forward to any updates from you!

Thanks.

mu...@gmail.com <mu...@gmail.com> #63Oct 25, 2023 05:27AM

Hello, Google Team (

Comment #56

),

I wanted to provide some updates on the situation:

#1. You're absolutely correct; the crash is occurring on Android 12 and 12L as well. I initially missed this because the occurrence rate is relatively low. I'll be sharing the data related to this with the relevant teams.
#2. The crash appears on certain users' devices and it's not reproducible consistently. Additionally, I wouldn't be able to identify ART updates on these user devices.

Furthermore, I've attached device details that I obtained from the Play Console. Hopefully, this information can be of assistance in addressing this issue.

These crashes are significantly impacting our users and key performance metrics. Your prompt assistance in resolving this issue would be greatly appreciated.

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

jo...@ticketmaster.co.uk <jo...@ticketmaster.co.uk> #64Oct 25, 2023 05:10PM

As implied in this comment

https://issuetracker.google.com/issues/301833859#comment61, won't the potential fixes not be availabe until Jan 2024? We're seeing the same issue (increased crashes since Sept 12th).

mu...@gmail.com <mu...@gmail.com> #65Oct 27, 2023 05:12AM

Hi Google team, Any update on this? Looking forward to your reply!

lo...@google.com <lo...@google.com> #66Oct 27, 2023 10:46PM

January 2024 ART module update is the earliest that I can make the fix available.

I wish there were reliable steps to reproduce this. In absence of that we can only get the fix on user devices and then monitor if the crashes go down or not.

qi...@trip.com <qi...@trip.com> #67Oct 30, 2023 01:54AM

comment #61

, we reproduced once on a Samsung device, and got the video and tombstone file, please see details in attachments.

tombstone_16

1.4 MB

Download

zm06cn1a0hifhmgog22C7.mp4

19 MB

Download

qi...@trip.com <qi...@trip.com> #68Oct 30, 2023 01:57AM

Please check the video at 00:56. You can see the page goes black, which is when the crash occurs.

ly...@gmail.com <ly...@gmail.com> #69Oct 30, 2023 03:27AM

Hello Google Team,
We are facing the same problem, here is the crash stack trace below I copied from Play Console
backtrace:
#00 pc 0x00000000002a96d0 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1720)
#01 pc 0x00000000002a5f00 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1188)
#02 pc 0x00000000002a10f4 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#03 pc 0x00000000003aa57c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+312)
#04 pc 0x00000000003a6500 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+540)
#05 pc 0x00000000003a5d38 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#06 pc 0x00000000003a5bbc /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#07 pc 0x00000000003a2a2c /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+72)
#08 pc 0x000000000031e514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#09 pc 0x00000000005f0698 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#10 pc 0x00000000005c35dc /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#11 pc 0x0000000000406128 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#12 pc 0x00000000003371a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#13 pc 0x000000000023ea64 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#14 pc 0x000000000054436c /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1600)
#15 pc 0x00000000000ba80c /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+228)
#16 pc 0x0000000000054120 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

ly...@gmail.com <ly...@gmail.com> #70Oct 30, 2023 03:31AM

[libart.so] void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&)

backtrace:
#00 pc 0x000000000024b250 /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&)+188)
#01 pc 0x000000000024b0a0 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1820)
#02 pc 0x0000000000247c7c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#03 pc 0x0000000000244640 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#04 pc 0x00000000003bde5c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#05 pc 0x00000000003bb8c0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#06 pc 0x00000000003bb114 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#07 pc 0x00000000003baf98 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#08 pc 0x0000000000393830 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#09 pc 0x000000000035d514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#10 pc 0x0000000000630a28 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#11 pc 0x000000000060396c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#12 pc 0x0000000000446428 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#13 pc 0x00000000003605a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#14 pc 0x000000000034b930 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#15 pc 0x00000000004f3e38 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#16 pc 0x00000000000e5a08 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#17 pc 0x000000000007f27c /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

so...@gmail.com <so...@gmail.com> #71Oct 31, 2023 04:36AM

Hi, google team
We detected many crash issue report from google play console as ConcurrentCopying crash or VisitReferences crash.

Application
targetSdkVersion 33
buildToolsVersion "33.0.2"
ndkVersion "22.1.7171670"
gradle:7.2.2

1. ConcurrentCopying
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

backtrace:
#00 pc 0x000000000024b03c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1720)
#01 pc 0x0000000000247c7c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#02 pc 0x0000000000244640 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#03 pc 0x00000000003bde5c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#04 pc 0x00000000003bb8c0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#05 pc 0x00000000003bb114 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#06 pc 0x00000000003baf98 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#07 pc 0x0000000000393830 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#08 pc 0x000000000032e514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#09 pc 0x0000000000604338 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#10 pc 0x00000000005d727c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#11 pc 0x0000000000419d38 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#12 pc 0x00000000003605a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#13 pc 0x000000000034b930 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#14 pc 0x00000000004f3e38 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#15 pc 0x00000000000b6668 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#16 pc 0x00000000000532cc /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

2. VisitReferences
backtrace:
#00 pc 0x00000000002a98e0 /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&)+184)
#01 pc 0x00000000002a9734 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1820)
#02 pc 0x00000000002a5f00 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1188)
#03 pc 0x00000000002a10f4 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#04 pc 0x00000000003aa57c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+312)
#05 pc 0x00000000003a6500 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+540)
#06 pc 0x00000000003a5d38 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#07 pc 0x00000000003a5bbc /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#08 pc 0x00000000003a2a2c /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+72)
#09 pc 0x0000000000330514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#10 pc 0x0000000000606508 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#11 pc 0x00000000005d943c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#12 pc 0x000000000041bf98 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#13 pc 0x00000000003371a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#14 pc 0x000000000023ea64 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#15 pc 0x000000000054436c /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1600)
#16 pc 0x00000000000ba5a8 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#17 pc 0x0000000000053e9c /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

lo...@google.com <lo...@google.com> #72Oct 31, 2023 11:44PM

Thanks for all the stacktraces. As I said in comment #61, I have a fix that will be cherry-picked into January ART module update. However, if any of you has any means to reproduce the issue then please let me know. It would be really helpful if you can share that with us. Or apply the fix and then try reproducing. But as I said, it's very important to have some reliability in reproduciblity.

22...@qq.com <22...@qq.com> #73Nov 1, 2023 06:52AM

Hello, Google team.
Can this problem be fixed in advance, or can users update the system version and roll back to before this patch package?
This is currently the case for many applications in our team. We are under pressure from our bosses and users. We may not be able to wait until January to update.
@ar...@google.com
@lo...@google.com
@ng...@google.com

id...@similarweb.com <id...@similarweb.com> #74Nov 1, 2023 09:01AM

Hello Google Team,
Joining the question in the last comment - Can you suggest a workaround until the fix will be released?
Can you also please share information on the nature of the fix or the suspected root cause?
Even if there is no "global" workaround, it might allow us to pinpoint functionality we might prefer to disable in the meanwhile.

22...@qq.com <22...@qq.com> #75Nov 2, 2023 01:18AM

lo...@google.com <lo...@google.com> #76Nov 2, 2023 01:31AM

The fix is in the GC'c code and rolling back will not be possible. And it's also not possible to make the fix available earlier than January. Another major problem is that since the issue is not reliably reproducible, we can't confirm that the fix will actually resolve the bugs that you are seeing.

What is the failure rate that you are seeing?

su...@oppo.corp-partner.google.com <su...@oppo.corp-partner.google.com> #77Nov 2, 2023 02:02AM

hi google team ,
The fix CL

comment #76

, is this one ?

https://android-review.googlesource.com/c/platform/art/+/2797139
We are working on this bug with app developer now , and with CL we still can reproduce this gc crash in monkey test

22...@qq.com <22...@qq.com> #78Nov 2, 2023 02:24AM

Our failure rate increased from 0.85% to 2.65% during the last two months recording to the Play Console.

js...@gmail.com <js...@gmail.com> #79Nov 2, 2023 07:03AM

Hi, our application's crash rate also increased from 0.7% to 1.6% and still increasing... :-( Google Play reports bad behavior with a red exclamation mark (User-perceived crash rate) and I guess they penalize us for that.

qu...@consulting-for.accor.com <qu...@consulting-for.accor.com> #80
Restricted
Nov 2, 2023 03:32PM

Comment has been deleted.

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

jo...@muzz.com <jo...@muzz.com> #81Nov 2, 2023 03:46PM

Our crash rate has followed the same pattern, increasing from Sept 12th, and has now stabilised at around 4.5% (user perceived crash rate). We do not have a repro scenario or more interesting crash logs to add, but thought it would be worth listing the libraries we use that include native libraries to see if many people are using the same libraries. Our app is using
- Realm
- Agora
- Yoti
- FFmpeg

lo...@google.com <lo...@google.com> #82Nov 3, 2023 12:15AM

Comment #77: thanks so much for trying. In that case I will keep looking for root cause of this bug.

lo...@google.com <lo...@google.com> #83Nov 3, 2023 12:17AM

Comment#77, can you please share the monkey testing process that you are following so that I can try locally?

su...@oppo.corp-partner.google.com <su...@oppo.corp-partner.google.com> #84Nov 3, 2023 09:42AM

hi google team,
We are very pleasure to share our monkey testing process , but as you know monkey test is just random test.. so maybe is meaningless..
Or perhaps , can we get some debug code for this bug ,as we nearly can reproduce 100% in our monkey test..

lo...@google.com <lo...@google.com> #85Nov 3, 2023 05:40PM

Hi Oppo team,

Good to know that you are able to reproduce this. Even if it's random test, it's very valuable. The most important thing is to be able to reproduce, no matter how. Because this allows us to add logs/debug info and then retry to track down the bug.

I can get you a debug ART module, but for that you will have to be using a userdebug build on the test phone. Is that possible for you?

I think it would be easier and faster if you give me steps to recreate your monkey test environment on my test device.

hb...@google.com <hb...@google.com> #86Nov 3, 2023 10:53PM

Clearly, finding a way to reproduce this and experiment would be great!

In the meantime is anybody in a position to tell whether the occurrence frequency here is correlated with the device type, e.g. whether this occurs more frequently with more modern cpu cores? If so, that would at least weakly suggest a memory ordering bug that's exposed by hardware that reorders more aggressively. Any more "invalid reference" crashes with information about the objects involved, like #30 above, would also be very interesting.

qi...@trip.com <qi...@trip.com> #87Nov 4, 2023 02:39AM

comment #86

, For running script to reproduce crash, I'v provided the tool, please see

comment #40

and

comment #54

. By doing so, you can reproduce this crash several times in 3 or 4 hours.

su...@oppo.corp-partner.google.com <su...@oppo.corp-partner.google.com> #88Nov 4, 2023 11:25AM

Comment#85 , yes ,we also can reproduce this bug on userdebug build .Its very kindof you can provide debug ART module. and is better that can provide debug patch, as we are build art from source code

For reproduce this bug ,as trip team Comment#87 .we are debugging this bug with crtip team together.And no matter ctrip or our, this reproduce must take a few hours..

lo...@google.com <lo...@google.com> #89Nov 4, 2023 01:36PM

OK. I will try to script from comment#40 again.

22...@qq.com <22...@qq.com> #90Nov 6, 2023 07:59AM

Hi google team,
Is there any new progress? Or with the necessary steps, how can we reduce the frequency of the problem? Hope to get the latest news on how it is progressing.
@hb...@google.com

lo...@google.com <lo...@google.com> #91Nov 6, 2023 07:27PM

I'm exactly using the steps written in how to run script.docx. The monkey-test attachment in comment#40 doesn't have few files/utils which the script seems to need:

iaet.properties
ct_apm_test_config.json
ibu-cli
scrcpy

Since these items are missing, I get the following error:

2023-11-06 10:56:31,246][INFO][__main__:234] [iaet] Download app Success
[2023-11-06 10:56:31,246][INFO][__main__:162] [iaet] lsof -i :27183
[2023-11-06 10:56:31,535][INFO][__main__:150] [iaet] scrcpy -s 21081FDH3001QH -b 2000000 --no-display -t --max-size 800 -p 27183 --record screen-record-21081FDH3001QH-ctrip.english-2023-11-06-10-48-46.mp4
[2023-11-06 10:56:31,537][INFO][__main__:378] run on device 21081FDH3001QH
/bin/sh: line 1: scrcpy: command not found
[2023-11-06 10:56:31,544][INFO][__main__:162] [iaet] adb -s 21081FDH3001QH shell am force-stop ctrip.english.debug
[2023-11-06 10:56:31,657][INFO][__main__:272] [iaet] Force-stop Result: Failure log like: 
[2023-11-06 10:56:31,657][INFO][__main__:162] [iaet] adb -s 21081FDH3001QH uninstall ctrip.english
[2023-11-06 10:56:31,737][INFO][__main__:281] [iaet] Uninstall Result: Failure log like: Failure [DELETE_FAILED_INTERNAL_ERROR]

[2023-11-06 10:56:31,737][INFO][__main__:162] [iaet] adb -s 21081FDH3001QH install -g /usr/local/google/home/lokeshgidra/code/to-delete/app-test/IBU_App_V7.86.5_10-18_20-30_Inhouse_PROD_18317653.apk
[2023-11-06 10:56:42,457][INFO][__main__:261] [iaet] Install Result: Success
[2023-11-06 10:56:42,457][INFO][__main__:157] [iaet] adb -s 21081FDH3001QH push ct_apm_test_config.json /sdcard/Android/data/ctrip.english/files/ct_apm_test_config.json
adb: error: cannot stat 'ct_apm_test_config.json': No such file or directory
[2023-11-06 10:56:42,471][INFO][__main__:157] [iaet] adb -s 21081FDH3001QH push iAET.jar /data/local/tmp/
iAET.jar: 1 file pushed, 0 skipped. 182.0 MB/s (845697 bytes in 0.004s)
[2023-11-06 10:56:42,509][INFO][__main__:157] [iaet] adb -s 21081FDH3001QH push iaet-preinput.json /sdcard/
iaet-preinput.json: 1 file pushed, 0 skipped. 2.5 MB/s (907 bytes in 0.000s)
[2023-11-06 10:56:42,552][INFO][__main__:157] [iaet] adb -s 21081FDH3001QH push iaet.properties /sdcard/
adb: error: cannot stat 'iaet.properties': No such file or directory
[2023-11-06 10:56:42,564][INFO][__main__:157] [iaet] adb -s 21081FDH3001QH push iaet-mapping.json /sdcard/
iaet-mapping.json: 1 file pushed, 0 skipped. 31.1 MB/s (17884 bytes in 0.001s)
[2023-11-06 10:56:42,681][INFO][__main__:157] [iaet] ibu-cli crbt -p ctrip.english -m EXPLORE_TEST -u 21081FDH3001QH
/bin/sh: line 1: ibu-cli: command not found

And since the above items are missing, I continuously get the following error:

2023-11-06 10:56:42,685][INFO][__main__:407] [21081FDH3001QH] start exploring
[2023-11-06 10:56:42,685][INFO][__main__:420] left running time 3 min
[2023-11-06 10:56:42,686][INFO][__main__:423] explore cmd: adb -s 21081FDH3001QH shell CLASSPATH='/data/local/tmp/iAET.jar' '/system/bin/app_process' '/data/local/tmp/iAET.jar' com.ibu.testing.Main -p ctrip.english --running-minutes 3 --support-crn --pre-input --module_list ibu-android-hotel
start explore
Options：-p ctrip.english --running-minutes 3 --support-crn --pre-input --module_list ibu-android-hotel 
processOptions：-p
processOptions：--running-minutes
processOptions：--support-crn
processOptions：--pre-input
processOptions：--module_list
[2023-11-06 10:56:42,968][INFO][__main__:157] [iaet] adb -s 21081FDH3001QH pull /sdcard/iAET-finish-tag.log result-21081FDH3001QH/iaet-finish-tag.log
adb: error: failed to stat remote object '/sdcard/iAET-finish-tag.log': No such file or directory

lo...@google.com <lo...@google.com> #92Nov 6, 2023 07:41PM

While the test was running, I continuously got the error but no activity on the phone screen. I followed all the steps of comment#54 as well:

I'm running the test on a pixel phone running Android 13
The device is unlocked during the execution of the script

I tried the script 2-3 times but still got the same error each time.

I think until the missing parts mentioned in comment #91 are resolved the script will not work. Please help to get it done asap.

lo...@google.com <lo...@google.com> #93Nov 7, 2023 02:46AM

Can anybody confirm if the crashes are visible more immediately after the user installs/updates the app, or even after 5-10 days of doing that?

su...@oppo.corp-partner.google.com <su...@oppo.corp-partner.google.com> #94Nov 7, 2023 03:28AM

Re comment #93
In our monkey environment , we can reproduce this bug by new install and running for hours.
btw, can you share us debug ART module as comment #85, by this ,we may be closer to root case

lo...@google.com <lo...@google.com> #95Nov 7, 2023 06:45AM

Unfortunately, it's not just one-time debug build that would be enough to get enough information. I will have to run keep adding/changing the log messages and keep reproducing. So the fastest would be for me to reproduce the crash. Kindly resolve the issues that I'm facing with monkey testing (comment#91).

Regarging my question about reproducing after fresh install vs. after few days, can you add a step in your monkey testing to see if the failure rate drops. Please compile the app before starting the monkey testing. I think you can do it using adb shell cmd package compile -f -m speed <package>

lo...@google.com <lo...@google.com> #96Nov 8, 2023 07:06AM

Any progress on getting the issues listed in comment#91 resolved? It's very important for me to be able to reproduce the issue for fixing it.

Also, did you try reproducing yourself but after compiling the app like described in comment#95?

If you really want to test on your end, then please apply this patch (https://android-review.googlesource.com/c/platform/art/+/2821430) and then build a debug module (instructions are in art/build/README.md). Also please use ART_HEAP_POISONING=true at build time. And try your script with this module to reproduce. Please provide any failure that you get with your app.

#apply the above CL
$banchan com.android.art.debug arm64
$export SOONG_ALLOW_MISSING_DEPENDENCIES=true BUILD_BROKEN_DISABLE_BAZEL=true
$ART_HEAP_POISONING=true m apps_only dist
$adb install out/dist/com.android.art.debug.apex
$adb reboot

qi...@trip.com <qi...@trip.com> #97Nov 8, 2023 07:43AM

comment #96

, we are working on it, will update the test tool to you later for reproducing crash.

qi...@trip.com <qi...@trip.com> #98
Restricted
Nov 8, 2023 09:49AM

Comment has been deleted.

deleted

Restricted

0 B

22...@qq.com <22...@qq.com> #99Nov 9, 2023 01:20AM

Hi google team,
Is there any new progress? Or with the necessary steps, how can we reduce the frequency of the problem? Hope to get the latest news on how it is progressing.

lo...@google.com <lo...@google.com> #100Nov 9, 2023 01:22AM

I tried running the new tool that you provided in comment#98 but I still get the same error. And I see that I still don't have the following files in the zip file:

ct_apm_test_config.json
iaet.properties
scrcpy (I don't need to record the screen when the crash happens. So I think this can be simply disabled)
In the new 'how to run scriptdoc you have said to install the trip.com, which installs packagectrip.englishbut the script force stopsctrip.english.debug`. This needs to be corrected.

qi...@trip.com <qi...@trip.com> #101Nov 9, 2023 03:59AM

Comment has been deleted.

Message last modified on Nov 9, 2023 04:00AM

deleted

0 B

qi...@trip.com <qi...@trip.com> #102
Restricted
Nov 9, 2023 04:00AM

Comment has been deleted.

deleted

Restricted

0 B

de...@similarweb.com <de...@similarweb.com> #103
Restricted
Nov 9, 2023 09:03PM

Comment has been deleted.

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

deleted

Restricted

0 B

de...@similarweb.com <de...@similarweb.com> #104Nov 9, 2023 09:19PM

In #103 the comment seems to have been restricted by mistake, while only attachments were meant to be restricted, I don't seem to be able to unrestrict it, feel free to make the comment's text public.

lo...@google.com <lo...@google.com> #105Nov 9, 2023 09:38PM

comment#102, thanks for the new version. This one actually works! But I haven't been able to reproduce so far.

One thing I noticed is that even if I use --running-minutes 60, the script finishes in 15-20 minutes. Why doesn't it run for the given amount of time?

su...@oppo.corp-partner.google.com <su...@oppo.corp-partner.google.com> #106Nov 10, 2023 03:22AM

Re comment#96, we also can reproduce this with speed mode. And with https://android-review.googlesource.com/c/platform/art/+/2821430 ,we also reproduce .Do you need log or coredump file? if needed,can i updated this to case 286023543

qi...@trip.com <qi...@trip.com> #107Nov 10, 2023 03:39AM

comment #105

, cause don't know what kind of issue you encountered specifically, this version is a temporary modification, so unexpected problems may occur. Please modify digital of "--running-minutes" to extend the time, the crash can take hours to reproduce.

lo...@google.com <lo...@google.com> #108Nov 10, 2023 04:21AM

Re comment#106, did you build the module with the CL in debug mode or not? Without debug the CL will have no effect. I can confirm this if you can just share the stacktrace here. If it has libartd.so then it's ok, otherwise you need to compile debug module.

lo...@google.com <lo...@google.com> #109Nov 10, 2023 04:21AM

I am using '--running-minutes' but when I give it large value, like 60, it still runs only for 10-15 mins and then finishes.

qi...@gmail.com <qi...@gmail.com> #110Nov 10, 2023 04:43AM

comment #109

, do you mean returning to the device's launcher and not calling the page again even if you wait a while?

In generally, if interrupted, wait about 10 seconds for it to call the page again, and so on until 60 minutes have passed.

lo...@google.com <lo...@google.com> #111Nov 10, 2023 04:54AM

I run with 60 minutes but the script finishes (I got back command prompt) in 20 mins.

su...@oppo.corp-partner.google.com <su...@oppo.corp-partner.google.com> #112Nov 11, 2023 02:28AM

Re comment#108, Sorry ,we cant reproduce this with libartd..as debug art bad performance ... the app anr beforce crash

22...@qq.com <22...@qq.com> #113Nov 11, 2023 10:21AM

Hi google team,
Is there any new progress?Our users have started to lose.

lo...@google.com <lo...@google.com> #114Nov 11, 2023 04:48PM

Re comment#113, which app is yours? It would be nice if you can also share steps to reproduce the issue on your app. It would be really helpful.

I have managed to reproduce the crash on trip.com app with the steps provided by their team and Oppo. I'm trying to figure out which last module was not having the issue. Simultaneously, I'm trying to figure out the right logs that hints me of the main cause of the problem.

The whole process will take some time.

de...@similarweb.com <de...@similarweb.com> #115
Restricted
Nov 13, 2023 05:00PM

Comment has been deleted.

deleted

Restricted

0 B

22...@qq.com <22...@qq.com> #116Nov 14, 2023 05:48AM

comment#114

,
We were unable to reproduce this on our own test equipment. Users just complain and leave without providing any valuable samples. At present, the problem can be reproduced. I hope it can be solved quickly. Come on.

jo...@muzz.com <jo...@muzz.com> #117Nov 14, 2023 10:24AM

Is this crash related to

https://github.com/getsentry/sentry-java/issues/2955? There seem to be multiple bug reports tracking similar things happening - do you think there are multiple bugs or just 1 change going on?

Our app does not use sentry but looking through the other native code one of the binaries is calling 'sigaltstack' (which sentry have removed in their beta release). Would other native code calling this function cause these kinds of issues? If so do you have any better explanation to how to change/alter this code to not cause these crashes?

ng...@google.com <ng...@google.com> #118Nov 14, 2023 10:33AM

The use of sigaltstack in Android is strongly discouraged, or must be tested with extreme care. The runtime already creates an alternate stack and makes sure it fits the needs of the signal handler.

So yes, we suggest changing the native code to remove the use of sigaltstack.

lo...@google.com <lo...@google.com> #119Nov 15, 2023 01:00AM

I have been trying to reproduce the bug with some logs enabled in ConcurrentCopying GC, but have failed to reproduce even after several hours. Can trip.com developers and Oppo try to reproduce with this CL https://android-review.googlesource.com/c/platform/art/+/2831135

It would be helpful if others who have some way of reproducing the bug to try with this CL and report here the crash-report. Thanks.

su...@oppo.corp-partner.google.com <su...@oppo.corp-partner.google.com> #120Nov 15, 2023 01:32AM

Re comment#119, We will try to reproduce with this cl ,and will update report as soon as we reproduce.

lo...@google.com <lo...@google.com> #121Nov 16, 2023 06:08PM

Hello, all of you who have any means to reproduce the crash on your respective apps, please try to reproduce with this CL: https://android-review.googlesource.com/c/platform/art/+/2832980.

Please let me know if the crashes are still there or gone. Thanks.

22...@qq.com <22...@qq.com> #122Nov 17, 2023 02:05AM

comment#121

Hello
How to use this script? Can we send it directly to the crashed user?

lo...@google.com <lo...@google.com> #123Nov 17, 2023 05:58AM

It's not a script. It's a patch for ART, which you will have to apply and then build and then try to reproduce. If you don't have the possibility of doing that, then you can ignore the comment. It was basically for those who can try to reproduce on their apps.

so...@google.com <so...@google.com> #124Nov 17, 2023 04:49PM

I prepared a CL with more debugging information (https://android-review.googlesource.com/c/platform/art/+/2835573). Can you patch it (and without Lokesh's CL) in and see a crash with the app? If you get logging information, can you post it here?

Again, if you don't have the possibility for patching ART feel free to ignore this comment.

da...@gmail.com <da...@gmail.com> #125Nov 20, 2023 03:18PM

the same issue: https://issuetracker.google.com/issues/173730326

qi...@trip.com <qi...@trip.com> #126Nov 21, 2023 10:06AM

Hi Google Team, as OPPO/MTK team have contacted you, and identified that this cl

https://android-review.googlesource.com/c/platform/art/+/2317296 related to this issue, what's your suggestions for application side to investigate the causes? maybe you can tell us the patterns of code writing may leads to this issue?

ng...@google.com <ng...@google.com> #127Nov 21, 2023 10:10AM

We're looking into it, and trying to identify such patterns.

22...@qq.com <22...@qq.com> #128Nov 24, 2023 02:49PM

Hi Google Team,
Our tests found that the problem of obvious memory leaks has been alleviated by off-band compression of resource files and some images within the App. Other developers can also try this to solve some problems. We imagine whether it is the September system patch that limits the system memory allocated by the App.

mb...@gmail.com <mb...@gmail.com> #129Nov 24, 2023 03:14PM

Re comment#128

Thanks for the interesting observation. What do you mean by off-band compression? Is it aapt's resource compression at build time? How did you fix it?

22...@qq.com <22...@qq.com> #130Nov 25, 2023 08:22AM

Re 129
For example, convert images in jpg format to webp format without bitmap deletion, etc.

22...@qq.com <22...@qq.com> #131Nov 28, 2023 02:25AM

Hi google team,
Is there any new progress? Hope to get the latest news on how it is progressing.

so...@google.com <so...@google.com> #132Nov 28, 2023 12:35PM

What we are doing right now is adding more debugging information (https://android-review.googlesource.com/c/platform/art/+/2831135) and trying to trigger the bug. Lokesh mentioned that it won't appear as a crash, but as an ANR. ANRs (when they happen) are located in /data/anr folder. If you open an ANR it will tell you which processes triggered it.

ANRs related to this bug will have held mutexes= "mutator lock"(exclusive held) in them. If you can upload them here, we can also take a look at the ANRs you encounter.

so...@google.com <so...@google.com> #133Nov 28, 2023 01:36PM

Oh and to repro Lokesh said to make sure we are using the CC GC. We can do that by

adb shell device_config set_sync_disabled_for_tests persistent
adb shell device_config put runtime_native_boot force_disable_uffd_gc true
adb reboot
# Make sure the CC GC is running
adb logcat |grep "concurrent copying"

22...@qq.com <22...@qq.com> #134Dec 5, 2023 01:08AM

Hi google team,
Asked when a repair patch would be released to completely solve this crash. After checking online, the user's crash situation became much more serious.

tw...@whova.com <tw...@whova.com> #135Dec 5, 2023 01:41AM

Hi, has there been any progress on this issue? We've also observed a trend of user-perceived-crash rates increasing up to ~0.75% in the days following a new version or when increasing our rollout percentage. We're trying to reproduce the issue as well, let me know if there's anything we can try or which files would be useful if we're able to trigger it.

Our app is also using `Agora` and `fmpeg-kit` which was mentioned by some other developers earlier in the thread

Thanks!

so...@google.com <so...@google.com> #136Dec 5, 2023 02:01PM

Re comment#134 we already submitted a fix which will be in the January train (there's no December train). That's the earliest we are able to push a fix to the users.

Re comment#135 we don't know the root cause of the bug. If you know how to patch this CL (https://android-review.git.corp.google.com/c/platform/art/+/2831135) to generate a new module / Android image do that. After you've done that, you can run:

adb shell device_config set_sync_disabled_for_tests persistent
adb shell device_config put runtime_native_boot force_disable_uffd_gc true
adb reboot
# Make sure the CC GC is running
adb logcat |grep "concurrent copying"

When you have that setup, you can upload the ANRs (/data/anrs) or tombstones (/data/tombstones) you encounter with your app to this bug report.

22...@qq.com <22...@qq.com> #137Dec 6, 2023 01:58AM

comment#136

Hi google team,
Can this repair be done in advance? We really need it. I am currently being complained about by users every day. Also, can this fix completely solve this problem 100%. Thank you.

so...@google.com <so...@google.com> #138Dec 6, 2023 11:53AM

Can this repair be done in advance?

January is the earliest we could do as there's no December train.

can this fix completely solve this problem 100%

On local testing by Googlers and by partners, the crash disappears after the CL that disables Write Barrier Elimination.

22...@qq.com <22...@qq.com> #139Dec 7, 2023 01:14AM

comment#138

Hi google team,

Thanks for your reply, we're really looking forward to this fix.
When can I update to this patch in January?

so...@google.com <so...@google.com> #140Dec 7, 2023 12:36PM

The devices will automatically update with the mainline January train

tw...@whova.com <tw...@whova.com> #141Dec 9, 2023 04:56PM

Thanks for the updates!

Our app is approaching the bad-behavior threshold from this crash, which will show a warning on our play store page. Is there a way to get an exemption until after the patch is rolled out to most users?

22...@qq.com <22...@qq.com> #142Dec 10, 2023 03:03AM

Agree with the question on

comment#141

.
Is there a way to get an exemption until after the patch is rolled out to most users?

so...@google.com <so...@google.com> #143Dec 11, 2023 05:23PM

Reassigned to ra...@google.com.

From the ART team we don't have the power to do that. It would be up to the Play team.

randeepsingh@google.com do you know someone who can assist with this? As a quick summary: app developers are seeing a crash due to an Android Runtime optimization. The fix is already in the pipeline and will be ready to users in January. The developers are being penalized due to hitting the crash threshold (App quality) and are asking if they can get a temporary exemption until the fix is live.

ds...@whova.com <ds...@whova.com> #144Dec 11, 2023 06:14PM

Hi google team,

Thanks a lot for the work you're putting into fixing this. Based on your investigation of the crash, are there any suggestions for changes we could make in our code to reduce the frequency of the crash until the fix is released in January?

Thanks

ds...@whova.com <ds...@whova.com> #145Dec 12, 2023 06:02PM

Hi,

I just want to follow up on

comment#143

. As previously mentioned by others, our app is approaching the bad behavior threshold due to this crash. This which will trigger a warning for our app on the play store as well as reduced visibility. Is there anyway for us to receive an exemption until the fix can be rolled out.

Thanks for the consideration

gi...@indiamart.com <gi...@indiamart.com> #146Dec 13, 2023 05:43AM

hi, because of this crash our app has crossed the bad behaviour threshold on play console, kindly provide a solution for this.

so...@google.com <so...@google.com> #147Dec 13, 2023 12:27PM

Reassigned to so...@google.com.

To provide an update, we started the discussion internally with the Play Store team. We will provide an update when we reach a decision. Thank you for your patience.

th...@threema.ch <th...@threema.ch> #148Dec 14, 2023 02:41PM

Because of lots of different native crashes which we cannot do anything about and seem to be related to gc problems, our apps ("Threema *" also crossed the bad behaviour threshold.
It is not exactly this same bug, but there is another ticket in the tracker:

https://issuetracker.google.com/issues/302680512
Thanks for including us in your considerations regarding bad visibility exemptions.

ds...@whova.com <ds...@whova.com> #149Dec 18, 2023 08:37PM

Hi Google Team,

We greatly appreciate that the Play Store team is taking time to discuss this issue. We're wondering if there's any update as to when we'll know whether or not we can be exempt from the Play Store bad behavior threshold policy.

Thanks a lot

ts...@gmail.com <ts...@gmail.com> #150Dec 19, 2023 02:37AM

Dear Google Play Support Team,

I hope this message finds you well. I am writing to address a critical issue that my application is currently experiencing, which is also impacting other apps on the Google Play platform.

Recently, my application has been suffering from a high crash rate, as evidenced in the Google Play Console. This issue is not only affecting the app's performance but also its standing on the platform. I am concerned about the potential penalties that my app might face due to these elevated crash rates.

I would like to request that my application not be penalized for this crash issue. It is important to note that this is a widespread problem affecting multiple applications, and it is not specific to the coding or design of my app.

Furthermore, I am eager to work towards a resolution. The frequent crashes have led to a significant increase in user complaints, which is understandably distressing for both the users and our team. We are committed to providing a high-quality user experience and would appreciate any guidance or assistance you can offer to help us resolve this issue promptly.

Thank you for your understanding and support in this matter. I look forward to your prompt response and am hopeful for a cooperative solution that benefits all parties involved.

Best regards,
Trans Team

tw...@whova.com <tw...@whova.com> #151Dec 22, 2023 07:57PM

Hi Google Team, we were wondering if there were any updates or progress made in the last week for the Google Play exemptions

Best,
Tony

gi...@indiamart.com <gi...@indiamart.com> #152Dec 27, 2023 06:12AM

hi, are there any updates

22...@qq.com <22...@qq.com> #153Dec 28, 2023 05:13AM

Hi google team,
When can I update to this patch in January?

gl...@whova.com <gl...@whova.com> #154Dec 29, 2023 01:13AM

Hi Google Team,

Is there any update for the bad visibility exemptions and the fix?

Best regards,
Guangda

22...@qq.com <22...@qq.com> #155Jan 2, 2024 10:02AM

Hi google team,
When can I update to this patch in January?

qi...@trip.com <qi...@trip.com> #156Jan 2, 2024 11:27AM

comment #140

Hi Google Team,

Happy new year and wish you all well!

You said earlier that a fix for the problem will be released in January, is the date confirmed yet?

so...@google.com <so...@google.com> #157Jan 2, 2024 02:09PM

Hello all,

I reached out to the play store teams. They are discussing alternatives but they haven't replied with any action items yet. I will update this bug report if I have some news.

The fix will be live for users with the January mainline train which will start to roll out to users next week. All mainline trains roll out to users over a few weeks. All users which get the update will do so by the end of the month.

tw...@whova.com <tw...@whova.com> #158Jan 3, 2024 05:50PM

Thanks for the update!

To clarify, which users will receive the update? Are they specific devices/OEMs (Pixel, Samsung, ...)?

Best,
Tony

ow...@gmail.com <ow...@gmail.com> #159Jan 3, 2024 05:59PM

Re comment #158

Google play system updates are oem agnostic. Any devices from android 10 onwards should receive the update, provided that they are otherwise eligible for GPSU.

so...@google.com <so...@google.com> #160Jan 3, 2024 07:10PM

Thanks ownerinl for the explanation. The Android Runtime in particular started being a mainline module in Android 12 so it is Android 12+ users, but the rest of your comment stands.

The January mainline train is able to reach all users that are in need of the fix.

tw...@whova.com <tw...@whova.com> #161Jan 4, 2024 11:31PM

Thanks for the clarification!

qi...@trip.com <qi...@trip.com> #162Jan 8, 2024 02:46AM

Hi Google Team，

Has the rollout started please? Please let us know if there are any updates.
Thanks.

so...@google.com <so...@google.com> #163Jan 8, 2024 11:59AM

Rollout will start this week and is scheduled to arrive to all eligible users by the end of January.

qi...@trip.com <qi...@trip.com> #164Jan 12, 2024 02:25AM

comment #163

. We've been monitoring online crash behavior since Monday of this week, and there's been no change in the crash trend. Has the rollout started or not please?
Thanks~

qi...@trip.com <qi...@trip.com> #165Jan 12, 2024 03:21AM

One more question, according to

comment #160

, "The January mainline train is able to reach all users that are in need of the fix." Do you mean that all users will be automatically updated with this patch and don't need to go into Google play to do it manually?

so...@google.com <so...@google.com> #166Jan 12, 2024 12:21PM

Rollout started on Wednesday. We estimate to reach 99% of users by Wednesday 17th.

Do you mean that all users will be automatically updated with this patch and don't need to go into Google play to do it manually?

The updates will happen automatically. The users can check if they have it by checking the Google Play System Updates on their phone (inside Settings). If it says January 2024, then they have the fix.

vi...@bigbasket.com <vi...@bigbasket.com> #167Jan 17, 2024 10:59AM

We've been monitoring online crash behavior since Monday of this week, and there's been no change in the crash trend. Do we know how many users are updated to this patch ?

js...@gmail.com <js...@gmail.com> #168Jan 17, 2024 11:06AM

For us, so far it's similar and the crashes are still at the same level.

so...@google.com <so...@google.com> #169Jan 17, 2024 03:28PM

Apologies, I was wrong with We estimate to reach 99% of users by Wednesday 17th. (I misread our internal document).
The estimate is that we will reach 100% of users by January 29th.

vi...@gmail.com <vi...@gmail.com> #170Jan 19, 2024 07:30AM

Hello Google Team,

Will the system update be installed automatically on the user's device, or does the user need to manually install the update by accessing the phone settings?

so...@google.com <so...@google.com> #171Jan 19, 2024 10:00AM

They should happen automatically, more info in this link's Get security updates & Google Play system updates section. The Android runtime update will happen through a Google Play system update.

22...@qq.com <22...@qq.com> #172Jan 20, 2024 01:57AM

Hello Google Team,
Whether these patches can effectively solve the crash issue, our crash indicators are not responding.

vi...@bigbasket.com <vi...@bigbasket.com> #173Jan 22, 2024 04:14AM

There is no change in crash trend after 12 days of release. Is the patch reaching to the users ?

so...@google.com <so...@google.com> #174Jan 22, 2024 08:50AM

The patch is reaching users. The rollout is estimated to reach 100% of users by January 29th. Not all users have the patch as of today.

be...@gmail.com <be...@gmail.com> #175Jan 22, 2024 10:15PM

Is there a way to manually install the January 2024 update? I check for Google Play System updates in the device settings but it says November 2023 is the latest available.

so...@google.com <so...@google.com> #176Jan 23, 2024 09:50AM

If you followed my link in comment#171 and you only see the November one, you will have to wait a bit longer. It is rolling out to users and in the coming days it should be available to you. Thanks for your patience

22...@qq.com <22...@qq.com> #177Jan 25, 2024 01:14AM

Hello Google Team,
Since the patch was released, crash indicators have not improved significantly, but have continued to worsen. We have notified users to update the patch, but the current effect is not ideal. Please confirm if the patch fixes this crash.

so...@google.com <so...@google.com> #178Jan 25, 2024 09:33AM

Are you seeing users with the January patch with the same errors as before? Or are they crashing for other reasons?

22...@qq.com <22...@qq.com> #179Jan 26, 2024 01:36AM

comment#178

There has been no feedback from users that the patch has actually been updated, but since the patch was released, the crashes have intensified and the content remains the same as before. We have no update features during this period.

22...@qq.com <22...@qq.com> #180Jan 27, 2024 05:26AM

It has just been confirmed that a user has updated the system patch, but the crash continues and the effect is not ideal.

de...@gmail.com <de...@gmail.com> #181Jan 28, 2024 10:47PM

We have similar crashes in our app

de...@gmail.com <de...@gmail.com> #182Jan 28, 2024 10:55PM

Could you please provide the name of the patch?

da...@gmail.com <da...@gmail.com> #183Jan 29, 2024 07:04AM

Are there other then 5.15, 5.10 kernels broken to exclude the Speculative Page Faults MM Code Exploit?

so...@google.com <so...@google.com> #184Jan 29, 2024 08:54AM

Re comment#180

Can you confirm that the user had the Google Play System Update from January 2024, and it is seeing the same crash? If so, can you open a new bug report and link it here? Thanks!

Re comment#181 and comment#182

The name of the patch is the Google Play System Update from January 2024. Users can check if they have it using the info in this link's Get security updates & Google Play system updates section. When users start taking the January train (this is done automatically for them) you should stop seeing these crashes.

Re comment#183

Lokesh, can you assist on this question?

vi...@bigbasket.com <vi...@bigbasket.com> #185Jan 29, 2024 09:31AM

Google Team,

We have not observed even a slight dip in number of crashes in this month. Is it possible that patch has not reached these users ?

Please note that we are not able to reproduce this at our end and neither in a position to connect to our users.

so...@google.com <so...@google.com> #186Jan 29, 2024 09:55AM

It is possible. Some users got the patch, and the January train has been halted (due to reasons unrelated with this bug). When it resumes, you will be able to see the dip. Thanks again for your patience

vi...@bigbasket.com <vi...@bigbasket.com> #187Jan 29, 2024 10:02AM

Thanks for the update.
Any ETA for this to resume ?

qi...@trip.com <qi...@trip.com> #188Jan 29, 2024 10:04AM

Hi Google Team,

Can you help to confirm with the rollout stage of the January train. So far we haven't observed a significant drop in crash either.

22...@qq.com <22...@qq.com> #189Jan 30, 2024 01:57AM

The fact proves that this January patch is of little use.

The problem of online user crashes is becoming increasingly severe, and all we can do is wait. After waiting for more than three months, the result is still the same. This is too disappointing. And I will also lose my job because of this.

qi...@trip.com <qi...@trip.com> #190Jan 30, 2024 02:26AM

comment#186

When will the train resume rollout? what's the plan from google side on resuming the rollout. And what's the stage of the rollout(the percentage of phase release) when it has been halted.

Thanks, looking forward for your reply.

jo...@muzz.com <jo...@muzz.com> #191Jan 30, 2024 08:41AM

We are also not seeing a reduction in crashes.
Are you able to give us an update about getting the play store to not downrate our apps because of the high crash rate? This will be helpful going forwards if the January release is not effective in stopping these crashes.

so...@google.com <so...@google.com> #192Jan 30, 2024 09:46AM

Re comment#187 and comment#188

I don't have an ETA to users getting the January update. The new January rollout has been deployed to a small number of users Friday 26th, and will ramp up when it looks stable.

Re comment#189

I am sorry to hear that, but unfortunately this is out of my hands. As soon as we identified the issue, we sent a fix. Given that there is no December train (per policy) and the January train got delayed, the fix hasn't reached users yet.

Re comment#190

I don't have access to the exact numbers but the general idea is to roll it out to a small number of users (< 0.1%) then wait a few days, ramp up to ~0.1%, monitor a bit more, and finally ramp up to 100% of users. Based on the timeline I'd say that more than 0.1% of users got the January update but I cannot say how many.

Re comment#191

I don't have any updates from the PlayStore side. Just to clarify, the January release was tested internally and with partners and it proved that it solves this issue.

lo...@google.com <lo...@google.com> #193Jan 30, 2024 10:29AM

comment#183 what do you mean? Why do you think Speculative page fault is broken? Or 5.10/5.15 kernels have any issue?

qi...@trip.com <qi...@trip.com> #194Jan 30, 2024 10:45AM

comment#192

Do you mean the January train delayed and was released on Friday 26th and has been halting so far? Do Google Team got a resuming rollout plan for January trains?
If you do got a plan, could you give us an approximate timeline for it? Maybe the rollout program is not your responsibility, can you help to ask, after all, this fix is very important for us!

Thanks

da...@gmail.com <da...@gmail.com> #195Jan 31, 2024 07:20AM

comment#193 they said https://vulners.com/zdt/1337DAY-ID-38248

so...@google.com <so...@google.com> #196Jan 31, 2024 09:38AM

Re comment#194

I mean that the new January train started rolling out to users Friday 26th. The new train hasn't been halted.
I don't have concrete timelines but my best guess would be 100% of users by Feb 16th (three weeks after Friday 26th). Usually it is a small ramp up the first week, and then bigger ramp ups the following two weeks. I think we will have more information by next Monday.

lo...@google.com <lo...@google.com> #197Feb 1, 2024 12:02AM

comment#195, SPF related issues are handled by the kernel team. Alo, this bug is unrelated to that. This bug would have happened on any kernel.

vi...@bigbasket.com <vi...@bigbasket.com> #198Feb 6, 2024 04:35AM

Team,

Please update on the current adoption, if possible

so...@google.com <so...@google.com> #199Feb 6, 2024 12:30PM

The new January train that started rolling out to users Friday 26th is looking good so far. It is ramping up but still only a small portion of users will have the update as of today (less than 1%).

22...@qq.com <22...@qq.com> #200Feb 6, 2024 05:13PM

Hi Google Team,
Since the effect is good, can we release the full amount? We really need this patch update.

vi...@gmail.com <vi...@gmail.com> #201Feb 9, 2024 05:48AM

Hi Google Team,
By when we can expect the rollout of the fix to all eligible users?

so...@google.com <so...@google.com> #202Feb 9, 2024 09:48AM

I have good news! I got confirmation that the rollout finished successfully and all users are eligible for the update now.

Thanks all for your patience during this time

vi...@bigbasket.com <vi...@bigbasket.com> #203Feb 9, 2024 09:53AM

Thanks for the update, team !

But does this also ensure that users have the fix now, or they are eligible and fix will reflect once they get the update ?

so...@google.com <so...@google.com> #204Feb 9, 2024 10:30AM

They will have the fix when they do the update. The users can force to check an update and install it, but this is not necessary as the phone automatically checks for updates. You should start seeing the crashes relating to this bug to go down soon.

so...@google.com <so...@google.com> Feb 12, 2024 05:17PM

Marked as fixed.

tw...@whova.com <tw...@whova.com> #205Feb 22, 2024 09:54PM

Thanks for the updates! Our user-perceived crash went down to ~0.4% before the fix was rolled out (likely due to overall traffic). Since February 9th, our crash-rate has trended downwards but by under 0.1% (mostly due to the rolling-window).

We were wondering what the expected behavior is for the Google Play System Updates. Does it get downloaded and installed automatically at some point?

From our devices, only one Pixel phone received the Google Play System Update automatically. For our other devices (various samsung phones) we were able to see the update but had to manually trigger the update more than a week after the update was available.

so...@google.com <so...@google.com> #206Feb 23, 2024 08:17AM

The Google Play System Updates are automatically downloaded and installed. There's a window where the app is available for download but your phone didn't do the Google Play System Updates download+install. In that window is when you can install it manually.

ds...@whova.com <ds...@whova.com> #207Feb 23, 2024 05:43PM

Hi Google Team,

Thanks for the additional information! Do you know how long this window where the update is available but the auto update hasn't occurred yet may last? The rollout started on January 26th and most of our devices have not received the auto update, so it seems like maybe it hasn't reached very many people yet.

Thanks

ds...@whova.com <ds...@whova.com> #208Mar 1, 2024 12:13AM

Hi Google Team,

Thank you for all of the updates you've been providing throughout this process. I'm just following up to see if there's any update on the percentage of users that should have gotten the auto update at this point. From our crash logs we can see that a good amount of users are still experiencing the crash.

Thanks

ds...@whova.com <ds...@whova.com> #209May 3, 2024 02:27AM

Hi Google Team,

We continue to see our users experiencing this crash. While it is less frequent than it was before, we still about 100 per week. The overwhelming majority (93%) of these crashes come from Samsung devices. Is it possible that the issue was not completely fixed or is there anyway for us to know if the people who are experiencing the crash just haven't downloaded the update yet?

Thanks a lot

lo...@google.com <lo...@google.com> #210May 3, 2024 06:05AM

Is it possible that some devices are still not updated with January 2024 ART module update? Is it possible for you to get the module version info on devices causing the crash? Also, can you share a stack trace of the crash?

jo...@muzz.com <jo...@muzz.com> #211May 3, 2024 07:51AM

We are seeing the same thing.
Our user perceived crash rate has come down a lot (from ~6% to now 1%) but seems to have got stuck there. Before the ART issue was introduced last year our user perceived crash rate was consistently around 0.2%. Crashlytics is currently reporting a crash free user rate of 99.55% (this does not include any native crashes).

All of the stack traces for the crashes are the same as they were before the rollout of the ART fix. We can't tell from our users if they have the ART module update installed - does Google not have stats for the uptake of each rollout? If the crash rate change is anything to go by it would suggest that ~10% of our users do not have the update installed.

All of these crashes showing up about 75% Android 14 and 25% Android 13, no crashes on other Android versions

Stack traces:
backtrace:
#00 pc 0x000000000005b6f0 /apex/com.android.runtime/lib64/bionic/libc.so (abort+168)
#01 pc 0x0000000000771b00 /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+904)
#02 pc 0x00000000000357d0 /apex/com.google.mainline.primary.libs@341177000/lib64/

libbase.so/75d3253827fcfd7a8d7b02ad45991611ec4ca424c0278e13e8acfad4d14e597a3ecff6c0caa2b785c73838528ee6e9c2b313240ff895f50ee39b1d7bc10f390a/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+80)
#03 pc 0x0000000000034d58 /apex/com.google.mainline.primary.libs@341177000/lib64/

libbase.so/75d3253827fcfd7a8d7b02ad45991611ec4ca424c0278e13e8acfad4d14e597a3ecff6c0caa2b785c73838528ee6e9c2b313240ff895f50ee39b1d7bc10f390a/libbase.so (android::base::LogMessage::~LogMessage()+352)
#04 pc 0x0000000000204a18 /apex/com.android.art/lib64/libart.so (art::gc::Verification::LogHeapCorruption(art::ObjPtr<art::mirror::Object>, art::MemberOffset, art::mirror::Object*, bool) const+2016)
#05 pc 0x00000000002afa9c /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&)+1072)
#06 pc 0x00000000002af578 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1820)
#07 pc 0x00000000002ad4ac /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1188)
#08 pc 0x00000000002a809c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#09 pc 0x00000000002a31e8 /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+312)
#10 pc 0x000000000029f5fc /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+548)
#11 pc 0x000000000046f6dc /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#12 pc 0x000000000046f560 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#13 pc 0x00000000003a2b08 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+72)
#14 pc 0x000000000033e514 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+116)
#15 pc 0x0000000000614f48 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#16 pc 0x00000000005e7c8c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Daemons$Daemon.run+172)
#17 pc 0x000000000042a578 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
#18 pc 0x000000000033eda4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#19 pc 0x0000000000239d54 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#20 pc 0x000000000053a1b0 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1600)
#21 pc 0x00000000000c3774 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#22 pc 0x000000000005d044 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

backtrace:
#00 pc 0x000000000024afd8 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1720)
#01 pc 0x0000000000247c18 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#02 pc 0x00000000002445dc /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#03 pc 0x00000000003be05c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#04 pc 0x00000000003bbac0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#05 pc 0x00000000003bb314 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#06 pc 0x00000000003bb198 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#07 pc 0x0000000000393a30 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#08 pc 0x0000000000010d14 /system/framework/arm64/boot-core-libart.oat (art_jni_trampoline+116)
#09 pc 0x0000000000049580 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+208)
#10 pc 0x000000000001f554 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$Daemon.run+180)
#11 pc 0x0000000000169ce0 /system/framework/arm64/boot.oat (java.lang.Thread.run+80)
#12 pc 0x00000000003607a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#13 pc 0x000000000034bac4 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#14 pc 0x00000000004f3fa4 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#15 pc 0x00000000000c37b4 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#16 pc 0x000000000005d084 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

backtrace:
#00 pc 0x00000000002af724 /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&)+184)
#01 pc 0x00000000002af578 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1820)
#02 pc 0x00000000002ad4ac /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1188)
#03 pc 0x00000000002a809c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#04 pc 0x00000000002a31e8 /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+312)
#05 pc 0x000000000029f5fc /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+548)
#06 pc 0x000000000046f6dc /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#07 pc 0x000000000046f560 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#08 pc 0x00000000003a2b08 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+72)
#09 pc 0x000000000000fd14 /system/framework/arm64/boot-core-libart.oat (art_jni_trampoline+116)
#10 pc 0x0000000000043758 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+200)
#11 pc 0x000000000001d18c /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$Daemon.run+172)
#12 pc 0x000000000015ead8 /system/framework/arm64/boot.oat (java.lang.Thread.run+72)
#13 pc 0x000000000033eda4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#14 pc 0x0000000000239d54 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#15 pc 0x000000000053a1b0 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1600)
#16 pc 0x00000000000c150c /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+204)
#17 pc 0x0000000000054930 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

ds...@whova.com <ds...@whova.com> #212May 3, 2024 01:31PM

https://issuetracker.google.com/issues/301833859#comment210: Thanks for the reply. From our logs, I don't think there's anyway for us to know if the devices have the update or not. Below are the stack traces for the crashes that we continue to see. The first is four times more frequent than the second and third.

1.
backtrace:
#00 pc 0x000000000024afd8 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1720)
#01 pc 0x0000000000247c18 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#02 pc 0x00000000002445dc /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#03 pc 0x00000000003be05c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#04 pc 0x00000000003bbac0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#05 pc 0x00000000003bb314 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#06 pc 0x00000000003bb198 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#07 pc 0x0000000000393a30 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#08 pc 0x0000000000010d14 /system/framework/arm64/boot-core-libart.oat (art_jni_trampoline+116)
#09 pc 0x0000000000049580 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+208)
#10 pc 0x000000000001f554 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$Daemon.run+180)
#11 pc 0x0000000000169ce0 /system/framework/arm64/boot.oat (java.lang.Thread.run+80)
#12 pc 0x00000000003607a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#13 pc 0x000000000034bac4 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#14 pc 0x00000000004f3fa4 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#15 pc 0x00000000000c37b4 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#16 pc 0x000000000005d084 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

2.
backtrace:
#00 pc 0x000000000024b1ec /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&)+188)
#01 pc 0x000000000024b03c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1820)
#02 pc 0x0000000000247c18 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#03 pc 0x00000000002445dc /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#04 pc 0x00000000003be05c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#05 pc 0x00000000003bbac0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#06 pc 0x00000000003bb314 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#07 pc 0x00000000003bb198 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#08 pc 0x0000000000393a30 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#09 pc 0x0000000000010d14 /system/framework/arm64/boot-core-libart.oat (art_jni_trampoline+116)
#10 pc 0x0000000000049580 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+208)
#11 pc 0x000000000001f554 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$Daemon.run+180)
#12 pc 0x0000000000169ce0 /system/framework/arm64/boot.oat (java.lang.Thread.run+80)
#13 pc 0x00000000003607a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#14 pc 0x000000000034bac4 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#15 pc 0x00000000004f3fa4 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#16 pc 0x00000000000c37f4 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#17 pc 0x000000000005d0c4 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

3.
backtrace:
#00 pc 0x000000000005b770 /apex/com.android.runtime/lib64/bionic/libc.so (abort+168)
#01 pc 0x000000000079120c /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+1380)
#02 pc 0x00000000000357d0 /apex/com.google.mainline.primary.libs@340737000/lib64/

libbase.so/5cbcb48dc4a1ceb71422e6e07e9f12bb952b969b1d4789234c07c3b6241482a6f3a09733800aa23496f3dcd787dd1ded12735fe6dc5b3049d050d96b769bc049/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_0::__invoke(char const*)+80)
#03 pc 0x0000000000034d58 /apex/com.google.mainline.primary.libs@340737000/lib64/

libbase.so/5cbcb48dc4a1ceb71422e6e07e9f12bb952b969b1d4789234c07c3b6241482a6f3a09733800aa23496f3dcd787dd1ded12735fe6dc5b3049d050d96b769bc049/libbase.so (android::base::LogMessage::~LogMessage()+352)
#04 pc 0x0000000000207724 /apex/com.android.art/lib64/libart.so (art::gc::Verification::LogHeapCorruption(art::ObjPtr<art::mirror::Object>, art::MemberOffset, art::mirror::Object*, bool) const+1976)
#05 pc 0x000000000024b6dc /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::ComputeLiveBytesAndMarkRefFieldsVisitor<true> const&)+1452)
#06 pc 0x000000000024b03c /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackForMarkingAndComputeLiveBytes()+1820)
#07 pc 0x0000000000247c18 /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1128)
#08 pc 0x00000000002445dc /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+184)
#09 pc 0x00000000003be05c /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#10 pc 0x00000000003bbac0 /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool, unsigned int)+508)
#11 pc 0x00000000003bb314 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool, unsigned int)+188)
#12 pc 0x00000000003bb198 /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+72)
#13 pc 0x0000000000393a30 /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+76)
#14 pc 0x0000000000010d14 /system/framework/arm64/boot-core-libart.oat (art_jni_trampoline+116)
#15 pc 0x0000000000049580 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+208)
#16 pc 0x000000000001f554 /system/framework/arm64/boot-core-libart.oat (java.lang.Daemons$Daemon.run+180)
#17 pc 0x0000000000169ce0 /system/framework/arm64/boot.oat (java.lang.Thread.run+80)
#18 pc 0x00000000003607a4 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612)
#19 pc 0x000000000034bac4 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144)
#20 pc 0x00000000004f3fa4 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1888)
#21 pc 0x00000000000c37f4 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
#22 pc 0x000000000005d0c4 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

ds...@whova.com <ds...@whova.com> #213May 3, 2024 04:41PM

Would also like to add that our app also includes the Agora, ffmpeg-kit, Zoom, and Jitsi dependencies (similar to what other developers have mentioned previously in this thread). The crash doesn't appear to be occurring when those dependencies are directly in use but not sure if their presence in the application could be having some impact.

gl...@whova.com <gl...@whova.com> #214May 3, 2024 05:44PM

Replying to

comment #50

and

comment #81

Our app (Whova) also uses the Agora SDK, and it uses version 3.7.0 (io.agora.rtc:full-rtc-basic:3.7.0). May I know the Agora version in your apps? (The Jeevansathi app and the Muzz app)

This might be useful to investigate the issue, thanks!

lo...@google.com <lo...@google.com> #215May 3, 2024 10:49PM

The problem is that this stack-trace only indicates memory corruption somewhere. It doesn't point to any specific issue in the runtime. On top of that, low reproducibility rate makes it almost impossible to debug. Earlier also we succeeded because ctrip app was able to provide a stress test to reproduce this. If anyone can provide a test to reproduce this reliably, then I can definitely try to debug. But without that it's almost impossible.

Which libraries/SDK are being used can be quite helpful because the memory corruption could be caused by one of them.

comment#211 and comment#212: can you please share complete trace and not just stacktrace? I want to look at the error message being logged or fault address, whatever the case maybe. There might be some useful hint available in that.

jo...@muzz.com <jo...@muzz.com> #216May 4, 2024 06:14AM

I've just copied the stack trace from the play console, so sadly I don't have the full trace to share.
It seems to me that either a good portion of our users haven't installed the January update (does Google have any stats around the uptake?), or something from the September ART change has made code that previously worked fine now crash (as was the previous issue but this looks like something different).

Are you able to look at data from the play console and see how many apps had a big difference in crash rate from September last year to January this year to now? To try and get an idea of how many apps are affected and whether this is a wide-spread problem?

Have you got any suggestions what kind of code may cause a crash now that wasn't causing a crash before? Is it likely to be native code or from Java/Kotlin code?

Re

comment #214

we no longer use agora in our app. When we removed it it didn't really make a difference to these crash reports.

ds...@whova.com <ds...@whova.com> #217May 7, 2024 06:23PM

Thanks for the reply. Re comment #215, is there a way for us to get the complete trace if we aren't able to reproduce the crash on our own devices?

da...@googlemail.com <da...@googlemail.com> #218Nov 15, 2024 09:15AM

Wanted to share my trace here. The error occurred on a Samsung XCover with Android 11.

It seems very similar to #30 and #55 (just to call 2 references of many).

Im I right and an Update to a newer Android version or Google Play Services/System (via Play store?) is the solution or what is the way to apply a fix? And if, which version if what is required?

This issue is marked as fixed but it is badly documented how it is fixed!

F/DEBUG   (25884): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
F/DEBUG   (25884): Build fingerprint: 'samsung/xcoverproeea/xcoverpro:11/RP1A.200720.012/G715FNXXU9CUJ3:user/release-keys'
F/DEBUG   (25884): Revision: '5'
F/DEBUG   (25884): ABI: 'arm64'
F/DEBUG   (25884): Timestamp: 2024-11-14 08:51:07+0100
F/DEBUG   (25884): pid: 6055, tid: 8368, name: HeapTaskDaemon  >>> de.brunata.android.gemo <<<
F/DEBUG   (25884): uid: 10269
F/DEBUG   (25884): signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
F/DEBUG   (25884): Abort message: '  Invalid reference: ref=0x30000000 referenced from: object=0x14e5c2b8 offset= 12
F/DEBUG   (25884):   obj=0x14e5c2b8 klass=0x71208160(java.lang.Class<android.view.Choreographer>) space=main space (region space) 0x12c00000-0x52c00000 card=112 adjacent_ram=2040b971000000300000000060005013000000000000501300000000606a5013|6081207100000030005a94700000003009000000000000000000000000000000
F/DEBUG   (25884):   obj->GetMarkBit()=1
F/DEBUG   (25884):   obj->GetReadBarrierState()=1
F/DEBUG   (25884):   Region containing obj:
F/DEBUG   (25884): Region[137]=0x14e40000-0x14e80000-0x14e80000 state=RegionStateAllocated type=RegionTypeUnevacFromSpace objects_allocated=7777 alloc_time=126 live_bytes=239048 ratio over allocated bytes=0.911896 longest_consecutive_free_bytes=18446744073709551560 (-56B) is_newly_allocated=false is_a_tlab=false thread=0x0
F/DEBUG   (25884):   region_space_bitmap_->Test(obj)=true
F/DEBUG   (25884):   ref=0x30000000 klass=0x0 <invalid address> space=main space (region space) 0x12c00000-0x52c00000 card=0 adjacent_ram=0000000000000000000000000000000000000000000000000000000000000000|0000000000000000000000000000000000000000000000000000000000000000
F/DEBUG   (25884):   ref->GetMarkBit()=0
F/DEBUG   (25884):   ref->GetReadBarrierState()=0
F/DEBUG   (25884):   Region containing ref:
F/DEBUG   (25884): Region[1872]=0x30000000-0x30000000-0x30040000 state=RegionStateFree type=RegionTypeNone objects_allocated=0 alloc_time=0 live_bytes=18446744073709551615 is_newly_allocated=false is_a_tlab=false thread=0x0
F/DEBUG   (25884):   region_space_bitmap_->Test(ref)=false'
F/DEBUG   (25884):     x0  0000000000000000  x1  00000000000020b0  x2  0000000000000006  x3  00000071222a6fb0
F/DEBUG   (25884):     x4  00000071ba1da000  x5  00000071ba1da000  x6  00000071ba1da000  x7  0000000000248034
F/DEBUG   (25884):     x8  00000000000000f0  x9  a254f9ffcfab1d4e  x10 0000000000000000  x11 ffffffc0fffffbdf
F/DEBUG   (25884):     x12 0000000000000001  x13 0000000000000336  x14 0000000000000000  x15 c95e3be050875e7d
F/DEBUG   (25884):     x16 00000071b4af8948  x17 00000071b4ad72d0  x18 00000070c9db6000  x19 00000000000017a7
F/DEBUG   (25884):     x20 00000000000020b0  x21 00000000ffffffff  x22 000000000000006e  x23 000000000000006e
F/DEBUG   (25884):     x24 0000007131dd3947  x25 0000000000000002  x26 0000007131deab5d  x27 00000071323f9000
F/DEBUG   (25884):     x28 00000071b8c21840  x29 00000071222a7030
F/DEBUG   (25884):     lr  00000071b4a8aca4  sp  00000071222a6f90  pc  00000071b4a8acd0  pst 0000000000000000
F/DEBUG   (25884): backtrace:
F/DEBUG   (25884):       #00 pc 0000000000089cd0  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 698b6aef520f034a9d40736d477f7a96)
F/DEBUG   (25884):       #01 pc 0000000000566728  /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+2308) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #02 pc 0000000000013978  /system/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_3::__invoke(char const*)+76) (BuildId: 7f81cc97d60164bfae3b52a2cd822f20)
F/DEBUG   (25884):       #03 pc 0000000000012fa4  /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+320) (BuildId: 7f81cc97d60164bfae3b52a2cd822f20)
F/DEBUG   (25884):       #04 pc 00000000002e7570  /apex/com.android.art/lib64/libart.so (art::gc::Verification::LogHeapCorruption(art::ObjPtr<art::mirror::Object>, art::MemberOffset, art::mirror::Object*, bool) const+1428) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #05 pc 000000000026d550  /apex/com.android.art/lib64/libart.so (void art::gc::collector::ConcurrentCopying::Process<true>(art::mirror::Object*, art::MemberOffset)+880) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #06 pc 000000000026c578  /apex/com.android.art/lib64/libart.so (void art::mirror::Object::VisitReferences<true, (art::VerifyObjectFlags)0, (art::ReadBarrierOption)1, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true>, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> >(art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&, art::gc::collector::ConcurrentCopying::RefFieldsVisitor<true> const&)+152) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #07 pc 0000000000276a24  /apex/com.android.art/lib64/libart.so (void art::gc::accounting::SpaceBitmap<8ul>::VisitMarkedRange<art::gc::collector::ConcurrentCopying::CopyingPhase()::$_8 const&>(unsigned long, unsigned long, art::gc::collector::ConcurrentCopying::CopyingPhase()::$_8 const&) const+800) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #08 pc 0000000000256420  /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::CopyingPhase()+484) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #09 pc 0000000000254308  /apex/com.android.art/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+1132) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #10 pc 0000000000278360  /apex/com.android.art/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+312) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #11 pc 000000000029535c  /apex/com.android.art/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool)+4156) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #12 pc 00000000002a7908  /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool)+124) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #13 pc 00000000002ad71c  /apex/com.android.art/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+36) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #14 pc 00000000002e683c  /apex/com.android.art/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+64) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #15 pc 000000000001260c  /apex/com.android.art/javalib/arm64/boot-core-libart.oat (art_jni_trampoline+124) (BuildId: 2ce37752b2a71c635a64328ec2e5387d37c221f9)
F/DEBUG   (25884):       #16 pc 0000000000059614  /apex/com.android.art/javalib/arm64/boot-core-libart.oat (java.lang.Daemons$HeapTaskDaemon.runInternal+196) (BuildId: 2ce37752b2a71c635a64328ec2e5387d37c221f9)
F/DEBUG   (25884):       #17 pc 0000000000024d50  /apex/com.android.art/javalib/arm64/boot-core-libart.oat (java.lang.Daemons$Daemon.run+160) (BuildId: 2ce37752b2a71c635a64328ec2e5387d37c221f9)
F/DEBUG   (25884):       #18 pc 000000000015ce98  /apex/com.android.art/javalib/arm64/boot.oat (java.lang.Thread.run+72) (BuildId: c3475b07d0e9fda8252c4d048848217b2ca92b3c)
F/DEBUG   (25884):       #19 pc 0000000000134564  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #20 pc 00000000001a9a78  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #21 pc 000000000055e278  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithJValues<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, jvalue const*)+460) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #22 pc 00000000005ada3c  /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1308) (BuildId: 1f074c499f06d21b85a268cca474631c)
F/DEBUG   (25884):       #23 pc 00000000000eb7a8  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+64) (BuildId: 698b6aef520f034a9d40736d477f7a96)
F/DEBUG   (25884):       #24 pc 000000000008bc8c  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 698b6aef520f034a9d40736d477f7a96)

so...@google.com <so...@google.com> #219Nov 15, 2024 09:21AM

Your error is memory corruption on Android R (the version is RP1A.200720.012) so it is unrelated to this bug. The bug started in Android U, and affected S+ devices.

r.android.com/2832980 disabled the optimization that caused the crash and was cherry picked so that users would stop seeing the crashes. The pass was later re-enabled in r.android.com/2926853 and it already reached users without causing issues.

Issue 301833859

Description

Issue summary

Comments

vi...@google.com <vi...@google.com> #2Sep 25, 2023 12:54PM

qi...@trip.com <qi...@trip.com> #3Sep 25, 2023 01:36PM

qi...@trip.com <qi...@trip.com> #4Sep 26, 2023 11:07AM

vi...@google.com <vi...@google.com> #5Sep 27, 2023 12:40PM

qi...@gmail.com <qi...@gmail.com> #6Sep 27, 2023 12:53PM

qi...@trip.com <qi...@trip.com> #7Sep 27, 2023 12:54PM

vi...@google.com <vi...@google.com> #8Sep 28, 2023 03:37PM

qi...@gmail.com <qi...@gmail.com> #9Sep 30, 2023 08:06AM

qi...@trip.com <qi...@trip.com> #10Sep 30, 2023 08:07AM

vi...@google.com <vi...@google.com> #11Oct 1, 2023 12:16AM

qi...@trip.com <qi...@trip.com> #12Oct 6, 2023 01:21PM

ng...@google.com <ng...@google.com> #13Oct 9, 2023 09:02AM

qi...@gmail.com <qi...@gmail.com> #14Oct 9, 2023 09:33AM

qi...@trip.com <qi...@trip.com> #15Oct 9, 2023 09:34AM

ng...@google.com <ng...@google.com> #16Oct 9, 2023 09:38AM

qi...@gmail.com <qi...@gmail.com> #17Oct 9, 2023 10:23AM

qi...@trip.com <qi...@trip.com> #18Oct 9, 2023 10:26AM

qi...@trip.com <qi...@trip.com> #19Oct 10, 2023 01:35PM

qi...@trip.com <qi...@trip.com> #20Oct 11, 2023 07:59AM

de...@similarweb.com <de...@similarweb.com> #21Oct 12, 2023 02:25PM

qi...@trip.com <qi...@trip.com> #22Oct 13, 2023 10:21AM

qi...@trip.com <qi...@trip.com> #23Oct 16, 2023 04:54AM

ng...@google.com <ng...@google.com> #24Oct 16, 2023 10:58AM

qi...@gmail.com <qi...@gmail.com> #25Oct 16, 2023 11:05AM

qi...@trip.com <qi...@trip.com> #26Oct 16, 2023 11:07AM

de...@similarweb.com <de...@similarweb.com> #27Oct 16, 2023 11:10AM

ng...@google.com <ng...@google.com> #28Oct 16, 2023 11:21AM

qi...@trip.com <qi...@trip.com> #29Oct 16, 2023 11:29AM

de...@similarweb.com <de...@similarweb.com> #30Oct 16, 2023 05:58PM

lo...@google.com <lo...@google.com> #31Oct 16, 2023 11:47PM

lo...@google.com <lo...@google.com> #32Oct 17, 2023 12:50AM

qi...@trip.com <qi...@trip.com> #33Oct 17, 2023 12:55AM

qi...@trip.com <qi...@trip.com> #34Oct 17, 2023 01:11AM

ng...@google.com <ng...@google.com> #35Oct 17, 2023 08:24AM

de...@similarweb.com <de...@similarweb.com> #36Oct 17, 2023 08:27AM

lo...@google.com <lo...@google.com> #37Oct 17, 2023 11:36PM

qi...@trip.com <qi...@trip.com> #38Oct 18, 2023 12:46AM

qi...@trip.com <qi...@trip.com> #39Oct 18, 2023 12:54PM

qi...@trip.com <qi...@trip.com> #40 Restricted Oct 18, 2023 01:22PM

qi...@trip.com <qi...@trip.com> #41Oct 19, 2023 07:43AM

lo...@google.com <lo...@google.com> #42Oct 19, 2023 11:39PM

qi...@trip.com <qi...@trip.com> #43Oct 20, 2023 02:25AM

mu...@gmail.com <mu...@gmail.com> #44Oct 20, 2023 07:58AM

mu...@gmail.com <mu...@gmail.com> #45Oct 20, 2023 07:59AM

mu...@gmail.com <mu...@gmail.com> #46Oct 20, 2023 08:00AM

lo...@google.com <lo...@google.com> #47Oct 20, 2023 06:05PM

lo...@google.com <lo...@google.com> #48Oct 20, 2023 06:06PM

lo...@google.com <lo...@google.com> #49Oct 20, 2023 06:07PM

mu...@gmail.com <mu...@gmail.com> #50Oct 21, 2023 07:33AM

mu...@gmail.com <mu...@gmail.com> #51Oct 21, 2023 07:34AM

mm...@gmail.com <mm...@gmail.com> #52Oct 21, 2023 03:09PM

vi...@gmail.com <vi...@gmail.com> #53Oct 23, 2023 08:13AM

qi...@trip.com <qi...@trip.com> #54Oct 23, 2023 09:08AM

qi...@trip.com <qi...@trip.com> #55Oct 23, 2023 10:22AM

lo...@google.com <lo...@google.com> #56Oct 23, 2023 11:23PM

lo...@google.com <lo...@google.com> #57Oct 23, 2023 11:24PM

lo...@google.com <lo...@google.com> #58Oct 23, 2023 11:32PM

qi...@trip.com <qi...@trip.com> #59Oct 24, 2023 07:06AM

qi...@trip.com <qi...@trip.com> #60Oct 24, 2023 07:10AM

lo...@google.com <lo...@google.com> #61Oct 25, 2023 01:03AM

qi...@trip.com <qi...@trip.com> #62Oct 25, 2023 03:55AM

mu...@gmail.com <mu...@gmail.com> #63Oct 25, 2023 05:27AM

jo...@ticketmaster.co.uk <jo...@ticketmaster.co.uk> #64Oct 25, 2023 05:10PM

mu...@gmail.com <mu...@gmail.com> #65Oct 27, 2023 05:12AM

lo...@google.com <lo...@google.com> #66Oct 27, 2023 10:46PM

qi...@trip.com <qi...@trip.com> #67Oct 30, 2023 01:54AM

qi...@trip.com <qi...@trip.com> #68Oct 30, 2023 01:57AM

ly...@gmail.com <ly...@gmail.com> #69Oct 30, 2023 03:27AM

ly...@gmail.com <ly...@gmail.com> #70Oct 30, 2023 03:31AM

so...@gmail.com <so...@gmail.com> #71Oct 31, 2023 04:36AM

lo...@google.com <lo...@google.com> #72Oct 31, 2023 11:44PM

22...@qq.com <22...@qq.com> #73Nov 1, 2023 06:52AM

id...@similarweb.com <id...@similarweb.com> #74Nov 1, 2023 09:01AM

22...@qq.com <22...@qq.com> #75Nov 2, 2023 01:18AM

lo...@google.com <lo...@google.com> #76Nov 2, 2023 01:31AM

su...@oppo.corp-partner.google.com <su...@oppo.corp-partner.google.com> #77Nov 2, 2023 02:02AM

qi...@trip.com <qi...@trip.com> #40
Restricted
Oct 18, 2023 01:22PM

qu...@consulting-for.accor.com <qu...@consulting-for.accor.com> #80
Restricted
Nov 2, 2023 03:32PM

qi...@trip.com <qi...@trip.com> #98
Restricted
Nov 8, 2023 09:49AM

qi...@trip.com <qi...@trip.com> #102
Restricted
Nov 9, 2023 04:00AM

de...@similarweb.com <de...@similarweb.com> #103
Restricted
Nov 9, 2023 09:03PM

de...@similarweb.com <de...@similarweb.com> #115
Restricted
Nov 13, 2023 05:00PM