IssueTracker

Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link:      https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

I've landed a change that will set the following properties also when the benchmark build type already exists:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

As well as the following for agp 8.0:

isDebuggable = false

I'm going ahead and closing this - if you've further questions please answer here and will reopen. Thanks.

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

Experiencing the same issue.

Stacktrace:

java.lang.IllegalStateException: Failed to stop [ProcessPid(processName=perfetto, pid=24965)]
at androidx.benchmark.Shell.terminateProcessesAndWait(Shell.kt:617)
at androidx.benchmark.Shell.terminateProcessesAndWait(Shell.kt:587)
at androidx.benchmark.perfetto.PerfettoHelper$Companion.stopAllPerfettoProcesses(PerfettoHelper.kt:470)
at androidx.benchmark.perfetto.PerfettoHelper.startCollecting(PerfettoHelper.kt:85)
at androidx.benchmark.perfetto.PerfettoCapture.start(PerfettoCapture.kt:69)
at androidx.benchmark.perfetto.PerfettoCaptureWrapper.start(PerfettoCaptureWrapper.kt:76)
at androidx.benchmark.perfetto.PerfettoCaptureWrapper.record(PerfettoCaptureWrapper.kt:131)
at androidx.benchmark.perfetto.PerfettoCaptureWrapper.record$default(PerfettoCaptureWrapper.kt:95)
at androidx.benchmark.macro.MacrobenchmarkPhaseKt.runPhase(MacrobenchmarkPhase.kt:105)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmark$4.invoke(Macrobenchmark.kt:258)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmark$4.invoke(Macrobenchmark.kt:255)
at androidx.benchmark.perfetto.PerfettoTraceProcessor$Companion.runServer-VtjQ1oo(PerfettoTraceProcessor.kt:133)
at androidx.benchmark.perfetto.PerfettoTraceProcessor$Companion.runServer(PerfettoTraceProcessor.kt:108)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:255)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:392)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:108)
at com.example.baselineprofile.StartupBenchmarks.benchmark(StartupBenchmarks.kt:52)
at com.example.baselineprofile.StartupBenchmarks.startupCompilationNone(StartupBenchmarks.kt:44)

Versions:

• androidx.benchmark:benchmark-macro-junit4:1.3.0
• AGP: 8.6.0
• Android Studio: Android Studio Koala Feature Drop | 2024.1.2

Device:

• Google Pixel 6 Pro
• Android 15
• Android security update: August 5, 2024
• Google Play system update: July 1, 2024
• Kernel version:
  5.10.210-android13-4-03254-g38965378c80a-ab11958173
  #1 Wed Jun 12 03:24:21 UTC 2024
• Build number: AP41.240726.010
• Not rooted

The full logs of two test runs can be found in the attachments of the previous comment.

This section in the logs seems interesting:

09-03 11:09:10.873 10922 10942 D PerfettoCapture: Supported ABIs: arm64-v8a, armeabi-v7a, armeabi
09-03 11:09:10.873 10922 10942 D PerfettoCapture: Recording perfetto trace
09-03 11:09:10.873 27764 10817 I NetworkScheduler.Stats: (REDACTED) Task %s/%s started execution. cause:%s exec_start_elapsed_seconds: %s
09-03 11:09:10.876 27764 10713 I NetworkScheduler.Stats: (REDACTED) Task %s/%s finished executing. cause:%s result: %s elapsed_millis: %s uptime_millis: %s exec_start_elapsed_seconds: %s
09-03 11:09:10.890  4184  5455 W MediaProvider: isAppCloneUserPair for user 0: false
09-03 11:09:10.895 10922 10942 I PerfettoCapture: Cleanup perfetto before starting.
09-03 11:09:10.934 27764 10817 I NetworkScheduler.Stats: (REDACTED) Task %s/%s started execution. cause:%s exec_start_elapsed_seconds: %s
09-03 11:09:10.938 27764 11369 I NetworkScheduler.Stats: (REDACTED) Task %s/%s finished executing. cause:%s result: %s elapsed_millis: %s uptime_millis: %s exec_start_elapsed_seconds: %s
09-03 11:09:10.966  4184 30886 W MediaProvider: isAppCloneUserPair for user 0: false
09-03 11:09:11.003  1109  1109 I WifiHAL : Creating message to get link statistics; iface = 47
09-03 11:09:11.019 27764 10817 I NetworkScheduler.Stats: (REDACTED) Task %s/%s started execution. cause:%s exec_start_elapsed_seconds: %s
09-03 11:09:11.023 27764 10820 I NetworkScheduler.Stats: (REDACTED) Task %s/%s finished executing. cause:%s result: %s elapsed_millis: %s uptime_millis: %s exec_start_elapsed_seconds: %s
09-03 11:09:11.045  4184 30886 W MediaProvider: Ignoring param:deletedata post U for external apps
09-03 11:09:11.048  4184 30886 I MediaProvider: Invalidating LocalCallingIdentity cache for package com.example.baselineprofile. Reason: owned_database_row_deleted:/storage/emulated/0/Android/media/com.example.baselineprofile/temporaryScript3723391827797421641.tmp
09-03 11:09:11.051  4184 30886 I MediaProvider: Deleted 1 0 0 0 0 0 0 items on external_primary due to com.android.shell
09-03 11:09:11.054 10922 10942 D Benchmark: kill -TERM command output - Output(stdout=, stderr=/storage/emulated/0/Android/media/com.example.baselineprofile/temporaryScript3723391827797421641.tmp[2]: kill: 24965: Operation not permitted
09-03 11:09:11.054 10922 10942 D Benchmark: )

It looks like PerfettoCapture is trying to kill an already running instance of perfetto, but it doesn't have the rights to do so (last log line).

Listing running perfetto processes on the device after the test run shows:

adb -d shell ps | grep perfetto
statsd       15678     1   10846896   4480 0                   0 S perfetto
statsd       24965     1   10814128   3208 0                   0 S perfetto

The process ID of the second process matches the ID in the Stacktrace above. So PerfettoCapture tried to stop this one, but didn't have permission to do so.

Trying to kill these process via adb is also not permitted:

$ adb -d shell kill 24965
/system/bin/sh: kill: 24965: Operation not permitted

$ adb -d shell kill 15678
/system/bin/sh: kill: 15678: Operation not permitted

The user that owns the process seems to be statsd. This looks like the Statsd module. It seems to be responsible for running the perfetto process that can't be killed.

This is likely what comment #9 is also pointing at.

I don't have any other hardware devices with me for comparison runs.

When running the test on an Emulator, the log shows this:

09-03 12:06:55.431 10498 10516 D PerfettoCapture: Supported ABIs: arm64-v8a
09-03 12:06:55.431 10498 10516 D PerfettoCapture: Recording perfetto trace
09-03 12:06:55.435  1479  8742 W MediaProvider: isAppCloneUserPair for user 0: false
09-03 12:06:55.436 10498 10516 I PerfettoCapture: Cleanup perfetto before starting.
09-03 12:06:55.445  1285 10327 I NetworkScheduler.Stats: (REDACTED) Task %s/%s started execution. cause:%s exec_start_elapsed_seconds: %s
09-03 12:06:55.446  1285 10578 I NetworkScheduler.Stats: (REDACTED) Task %s/%s finished executing. cause:%s result: %s elapsed_millis: %s uptime_millis: %s exec_start_elapsed_seconds: %s
09-03 12:06:55.453  1479  8742 W MediaProvider: isAppCloneUserPair for user 0: false
09-03 12:06:55.464  1285 10327 I NetworkScheduler.Stats: (REDACTED) Task %s/%s started execution. cause:%s exec_start_elapsed_seconds: %s
09-03 12:06:55.466  1285  3618 I NetworkScheduler.Stats: (REDACTED) Task %s/%s finished executing. cause:%s result: %s elapsed_millis: %s uptime_millis: %s exec_start_elapsed_seconds: %s
09-03 12:06:55.474  1479  2128 I MediaProvider: Invalidating LocalCallingIdentity cache for package com.example.baselineprofile. Reason: owned_database_row_deleted:/storage/emulated/0/Android/media/com.example.baselineprofile/temporaryScript2502516040961053018.tmp
09-03 12:06:55.474  1479  2128 I MediaProvider: Deleted 1 0 0 0 0 0 0 items on external_primary due to com.android.shell
09-03 12:06:55.475 10498 10516 I PerfettoCapture: Starting perfetto tracing with cmd: cat /storage/emulated/0/Android/media/com.example.baselineprofile/trace_config.pb | perfetto --background-wait -c - -o /data/misc/perfetto-traces/trace_output.pb
09-03 12:06:55.478  1479  8742 W MediaProvider: isAppCloneUserPair for user 0: false
09-03 12:06:55.484  1285 10327 I NetworkScheduler.Stats: (REDACTED) Task %s/%s started execution. cause:%s exec_start_elapsed_seconds: %s
09-03 12:06:55.486  1285  3618 I NetworkScheduler.Stats: (REDACTED) Task %s/%s finished executing. cause:%s result: %s elapsed_millis: %s uptime_millis: %s exec_start_elapsed_seconds: %s
09-03 12:06:55.484 11210 11210 W perfetto: type=1400 audit(0.0:455): avc:  denied  { write } for  path="/data/local/tmp/temporaryScript4626443941097950496.tmp_stderr" dev="dm-39" ino=65625 scontext=u:r:perfetto:s0 tcontext=u:object_r:shell_data_file:s0 tclass=file permissive=0
09-03 12:06:55.493 11211 11211 I perfetto:    perfetto_cmd.cc:1053 Connected to the Perfetto traced service, starting tracing
09-03 12:06:55.493   444   444 I perfetto: ng_service_impl.cc:1042 Configured tracing session 22, #sources:9, duration:0 ms, #buffers:2, total buffer size:36864 KB, total sessions:1, uid:2000 session name: ""
09-03 12:06:55.493   432   432 I perfetto:  probes_producer.cc:141 Ftrace setup (target_buf=43)
09-03 12:06:55.492   432   432 W traced_probes: type=1400 audit(0.0:456): avc:  denied  { read } for  name="current_tracer" dev="tracefs" ino=9295 scontext=u:r:traced_probes:s0 tcontext=u:object_r:debugfs_tracing_debug:s0 tclass=file permissive=0
09-03 12:06:55.493   432   432 I perfetto:    ftrace_procfs.cc:445 disabled ftrace in /sys/kernel/tracing/
09-03 12:06:55.708   432   432 W traced_probes: type=1400 audit(0.0:457): avc:  denied  { read } for  name="format" dev="tracefs" ino=8233 scontext=u:r:traced_probes:s0 tcontext=u:object_r:debugfs_tracing_debug:s0 tclass=file permissive=0
09-03 12:06:55.710   432   432 I perfetto:    ftrace_procfs.cc:442 enabled ftrace in /sys/kernel/tracing/
09-03 12:06:55.751  1479  6378 I MediaProvider: Invalidating LocalCallingIdentity cache for package com.example.baselineprofile. Reason: owned_database_row_deleted:/storage/emulated/0/Android/media/com.example.baselineprofile/temporaryScript4626443941097950496.tmp
09-03 12:06:55.753  1479  6378 I MediaProvider: Deleted 1 0 0 0 0 0 0 items on external_primary due to com.android.shell
09-03 12:06:55.754 10498 10516 I PerfettoCapture: Perfetto output - 11211
09-03 12:06:55.754 10498 10516 I PerfettoCapture: EXITCODE=0

Emulator info:

• Pixel 6 AVD
• Android 14
• Android security update: September 5, 2023
• Google Play system update: July 1, 2023
• Kernel version:
  6.1.23-android14-4-00257-g7e35917775b8-ab9964412
  #1 Mon Apr 17 20:50:58 UTC 2023
• Build number: UE1A.230829.036.A1 dev-keys

It seems like PerfettoCapture doesn't try to kill a perfetto process, and it seems to be able to start a perfetto trace.

Querying for an existing perfetto process on the emulator shows that no perfetto processes are running.

Hypothesis based on information in this thread:

On Pixel (maybe others too) devices with Android 15, the Statsd module is starting a perfetto process. This process can't be killed by PerfettoCapture when running a test, which produces the crash.

Edit 1: Seems like I hit a shortcut that sends the comment and deleted the text 🙈. Now I have to type it all again 😫. Hope I don't hit it again. Well, here we go 🤞.

Edit 2: Comment text got removed when posting the comment — and of course I didn't save it somewhere else. On to try number 3 🏇. Different browser...

Checked on a Pixel 6 Pro with Android 14 today. The crash did not occur.

I tried restarting the problematic device running 15, it did not resolve the issue. After a restart, the perfetto processes owned by statsd are also present again.

I also encountered this issue with the problematic Pixel 6 Pro running Android 15: https://issuetracker.google.com/issues/239897826. It also couldn't be resolved with a restart. On the Pixel 6 Pro with Android 14, recording a trace was possible.
Maybe the problem described in that issue could also be triggered by the perfetto process which can't be killed. The logs posted in this issue hint that the cause might also be a failure to run perfetto on the device.

I was able to occasionally reproduce this with my Pixel 5 with ToT on ScrollerBenchmark as well as TrivialKotlinBenchmark. Unfortunately I can't seem to reproduce this anymore, but here are some details anyway:

Device:

• Google Pixel 5
• Android 14

Project: platform/frameworks/support
Branch: androidx-main
Author: Chris Craik <ccraik@google.com>
Link:      https://android-review.googlesource.com/3463345

Avoid crashes from external perfetto trace capture

Avoid crashes from external perfetto trace capture 
 
Fixes: 323601788 
Test: ShellTest#killProcessesAndWait_failure 
Relnote: "Speculative fix for crashes: 'Failed to stop [ProcessPid(processName=perfetto, pid=...)]'. Now Benchmark will log a message instead of crashing when a backgroudn Perfetto process isn't able to be stopped before running the benchmark." 
 
From bugreport details, it appears that background perfetto traces are 
often triggered which cannot be terminated by shell. Log an error 
rather than crashing the benchmark. 
 
Change-Id: I37d3e254f7f3e49c53387dae89bb5bed7d2c9bf5

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-common:1.4.0-alpha08
• androidx.benchmark:benchmark-macro:1.4.0-alpha08