Status Update
Comments
di...@target.com <di...@target.com> #2
I had created the same issue with wrong component earlier
ra...@google.com <ra...@google.com>
ma...@google.com <ma...@google.com> #3
I noticed in the sample project you're using an emulator with api30.
We found a bug that is being fixed related to api 30 emulator and baseline profile generation.
Could you please try:
- Updating the baseline profile gradle plugin version (alpha14 is the latest as we speak)
- Change api version to 33 (or anything > 30)
Let me know if this works!
Thanks!
di...@gmail.com <di...@gmail.com> #4
di...@target.com <di...@target.com> #5
Thanks for suggestions and I could generate profiles in few scenarios. Few observations:
-
The issue seems to be specific to ARM devices running in FTL, regardless of the API version. When testing the application on a physical device (from FTL) with API 33, the profiles were successfully generated. This suggests that the problem lies in the ARM device profiles not being located where the extraction process expects them to be. It's worth investigating whether this issue originates from the plugin itself and whether it requires attention and resolution.
-
The current behavior of the
BaselineProfileConsumerExtension
does not save the optionssaveInSrc
andbaselineProfileOutputDir
in the designated locations. This inconsistency exists when running in FTL managed device. It would be beneficial to enhance the plugin's functionality to ensure consistent behavior across different environments. This improvement would ensure that the specified options for saving profiles are reliably honored, regardless of the testing environment being used.
ma...@google.com <ma...@google.com> #6
Thanks for your observations.
About your second point: The plugin does not make any distinction around what device the baseline profile generators run, when it come to saveInSrc
and baselineProfileOutputDir
. Could you please describe the issue more in detail? Or update your sample project if you'd like, so I can give a look.
ma...@google.com <ma...@google.com> #7
Actually could you also please check logcat for the devices where it wasn't generated and look for anything related to dex2oat
? I'm wondering if the issue is only in the emulators rather than the gradle task.
di...@target.com <di...@target.com> #8
I am trying to save it in the source after generating by following the
I have attached the sample project (Macrobenchmark) and included comments in the build.gradle.kts
file in baselineProfile
module.
di...@target.com <di...@target.com> #9
I re-checked the logs related to dex2oat
and could see this mismatch warning generated when it's running:
06-01 21:39:39.930: V/installd(463): Running /apex/com.android.art/bin/dex2oat64 in=base.apk out=/data/app/~~Sc5l67aRwmVS1G6tKcZP3w==/com.example.macrobenchmark.target-B8yQDmvRV9qvZZP0CfMurg==/oat/arm64/base.odex
...
06-01 21:39:39.941: W/dex2oat64(9628): Mismatch between dex2oat instruction set features to use (ISA: Arm64 Feature string: a53,crc,-lse,-fp16,-dotprod,-sve) and those from CPP defines (ISA: Arm64 Feature string: -a53,-crc,-lse,-fp16,-dotprod,-sve) for the command line:
06-01 21:39:39.941: W/dex2oat64(9628): /apex/com.android.art/bin/dex2oat64 --zip-fd=7 --zip-location=base.apk --oat-fd=8 --oat-location=/data/app/~~Sc5l67aRwmVS1G6tKcZP3w==/com.example.macrobenchmark.target-B8yQDmvRV9qvZZP0CfMurg==/oat/arm64/base.odex --input-vdex-fd=9 --output-vdex-fd=10 --app-image-fd=13 --image-format=lz4 --profile-file-fd=12 --swap-fd=11 --classpath-dir=/data/app/~~Sc5l67aRwmVS1G6tKcZP3w==/com.example.macrobenchmark.target-B8yQDmvRV9qvZZP0CfMurg== --class-loader-context=PCL[]{} --instruction-set=arm64 --instruction-set-features=default --instruction-set-variant=generic --compiler-filter=speed-profile --compilation-reason=cmdline --max-image-block-size=524288 --resolve-startup-const-strings=true --generate-mini-debug-info --runtime-arg -Xtarget-sdk-version:33 --runtime-arg -Xhidden-api-policy:enabled --runtime-arg -Xms64m --runtime-arg -Xmx512m
...
06-01 21:39:39.963: I/dex2oat64(9628): /apex/com.android.art/bin/dex2oat64 --input-vdex-fd=9 --output-vdex-fd=10 --profile-file-fd=12 --classpath-dir=/data/app/~~Sc5l67aRwmVS1G6tKcZP3w==/com.example.macrobenchmark.target-B8yQDmvRV9qvZZP0CfMurg== --class-loader-context=PCL[]{} --compiler-filter=speed-profile --compilation-reason=cmdline --max-image-block-size=524288 --resolve-startup-const-strings=true --generate-mini-debug-info
which is also present when running on non-ARM devices. I cannot say if this is an ARM specific issue or not, but this issue is prominent when running on ARM devices.
ma...@google.com <ma...@google.com> #10
I was looking for something like dex2oat
not found. This is what I suspect should happen on the ARM device api 30 where it doesn't work.
In the log you posted here it seems that it's working (although I'm not sure about that mismatch).
di...@target.com <di...@target.com> #11
Yes, I also couldn't see any such logs. Could you try the sample app attached and check?
di...@target.com <di...@target.com> #12
Any updates on this one? Will there be a fix in the BaselinesProfile plugin for handling FTL devices and ensuring the DSL works well with them?
ma...@google.com <ma...@google.com> #13
Seems like FTL doesn't support this currently but it's on their plan.
I'm not sure about timeline unfortunately.
di...@target.com <di...@target.com> #14
Thanks for the update. Do you also happen to know if BaselineProfiles plugin will support the results from FTL devices configured as GMD? Second point in my
ma...@google.com <ma...@google.com> #15
While you can configure FTL devices similarly to GMD, they are fundamentally different and they are managed by different tasks in gradle.
The baseline profile plugin can technically support FTL but it may not work until FTL gradle task support everything that the GMD task do.
ed...@gmail.com <ed...@gmail.com> #16
edmilsonpereira990@gmail.com
Em qui, 15 de jun de 2023 13:52, <buganizer-system@google.com> escreveu:
bu...@google.com <bu...@google.com> #18
di...@target.com <di...@target.com> #19
I am unable to access
ma...@google.com <ma...@google.com> #20
That task is internal.
This task can be fixed only when that one is completed.
za...@gmail.com <za...@gmail.com> #21
Could the internal issue be made public? It's not clear from the (sparse) docs I could find on managed devices and baseline profile generation that only GMDs were supported.
be...@google.com <be...@google.com> #22
Thanks for flagging this problem. If you're facing it, use the +1 button.
This issue will be added to the
li...@gmail.com <li...@gmail.com> #23
I've come to this issue from the list of known issues of Baseline Profiles plugin. There it says:
Baseline Profile generation isn't supported on Firebase Test Lab devices, including Gradle-managed Test Lab devices
However,
When testing the application on a physical device (from FTL) with API 33, the profiles were successfully generated.
So, does it mean that it's actually possible to generate baseline profile using FTL devices?
se...@team.bumble.com <se...@team.bumble.com> #24
FWIW, I can generate baseline profiles using FTL ARM device but without AndroidX Baseline Profile Gradle plugin. The FTL Gradle plugin currently doesn't honour instrumentation runner arguments (see
It would be great if AndroidX Baseline Profile and FTL plugins could finally work properly together, but I don't get my hopes up as the FTL plugin has seen very little development this year.
li...@gmail.com <li...@gmail.com> #25
FWIW, I can generate baseline profiles using FTL ARM device but without AndroidX Benchmark Gradle plugin.
Do you mean w/o androidx.baselineprofile
plugin? How do you run the benchmarks/fetch the profiles then? It would be awesome if you could share your setup.
bu...@google.com <bu...@google.com>
se...@team.bumble.com <se...@team.bumble.com> #26
com.android.test
), and tell FTL plugin to pull the contents of the package-specific media directory where the generated baseline profiles are saved.
plugins {
id("com.google.firebase.testlab")
}
testLab {
managedDevices{
create("firebaseSmallPhone") {
device = "SmallPhone.arm"
apiLevel = 34
}
}
testOptions {
results {
directoriesToPull.set(
listOf("/sdcard/Android/media/${android.namespace}")
)
}
}
}
At the end of the build, I copy the pulled baseline profile to the src directory:
find <path to baseline profile module>/build/outputs -name '*-baseline-prof.txt' \
-exec cp {} <path to app module>/src/main/baselineProfiles/baseline-prof.txt \;
li...@gmail.com <li...@gmail.com> #27
Thanks!
Could you please specify how you launch a baseline profile generator with this setup? Simply as an instrumentation test?
A little side note. I though that the whole issue is that the folder, where profiles are generated, is always different on an FTL device. With the above setup, what actually prevent you from using the Baseline Profile plugin?
ma...@google.com <ma...@google.com> #28
With aosp/3267102, the ftl plugin can now be used with the baseline profile gradle plugin as well. This will be added in version 1.3.2.
In order to use, simply add a baseline profile module, for example :baselineprofile
and add the following to build.gradle
:
plugins {
...
id("androidx.baselineprofile")
id("com.google.firebase.testlab")
}
...
firebaseTestLab {
// Credentials for FTL service
serviceAccountCredentials.set(file("credentials.json"))
// Creates one or more managed devices to run the tests on.
managedDevices {
"ftlDeviceShiba34" {
device = "shiba"
apiLevel = 34
}
}
// Ensures the baseline profile are pulled from the device.
// Note that this will be automated as well later with aosp/3272935.
testOptions {
results {
directoriesToPull.addAll(\"/storage/emulated/0/Android/media/${android.namespace}\")
}
}
}
baselineProfile {
...
managedDevices += "ftlDeviceShiba34"
useConnectedDevices = false
}
ap...@google.com <ap...@google.com> #29
Branch: androidx-main
commit 7e25ec644b2fd621b7a96016f9e397dab254540d
Author: Marcello Albano <maralb@google.com>
Date: Tue Sep 17 15:11:56 2024
Added support for FTL to Baseline Profile Gradle Plugin
FTL devices don't use per test artifacts, rather global ones,
so in test-results.pb they need to be found separately.
Note that the ftl plugin still needs to be configured separately.
Relnote: "Added support for FTL to Baseline Profile Gradle Plugin.
To use FTL apply the plugin to the baseline profile module in the
plugin block, with:
```
plugins {
...
id(\"com.google.firebase.testlab\")
}
```
and then configure firebase test lab with:
```
firebaseTestLab {
// Credentials for FTL service
serviceAccountCredentials.set(file(\"credentials.json\"))
// Creates one or more managed devices to run the tests on.
managedDevices {
\"ftlDeviceShiba34\" {
device = \"shiba\"
apiLevel = 34
}
}
// Ensures the baseline profile are pulled from the device.
// Note that this will be automated as well later with aosp/3272935.
testOptions {
results {
directoriesToPull.addAll(\"/storage/emulated/0/Android/media/${android.namespace}\")
}
}
}
```
Also the created ftl device needs to be added to the baseline profile extension:
```
baselineProfile {
managedDevices += \"ftlDeviceShiba34\"
useConnectedDevices = false
}
```"
Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 285187547
Change-Id: I2f6789d81a32d04df976437c2be426cbe04488ac
M benchmark/baseline-profile-gradle-plugin/src/main/kotlin/androidx/baselineprofile/gradle/producer/tasks/CollectBaselineProfileTask.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/consumer/BaselineProfileConsumerPluginTest.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/utils/BaselineProfileProjectSetupRule.kt
M benchmark/baseline-profile-gradle-plugin/src/test/kotlin/androidx/baselineprofile/gradle/utils/TestUtils.kt
li...@gmail.com <li...@gmail.com> #30
I apologise for posting an offtopic, but attempting to setup Baseline Profiles generation, using the above instructions and FTL Gradle plugin, I've encountered a strange dependency issue:
Firebase section of the issue tracker is not available for me, so posting a reference here with a hope, that somebody with access could check for duplicates and maybe suggest a solution.
ma...@google.com <ma...@google.com> #31
Thanks, the dependency issue is a different one already been tracked. Closing this one.
li...@gmail.com <li...@gmail.com> #32
Thanks!
Which version is this fix coming to? Will it be 1.3.2 or we'll need to wait for 1.4.0 release?
pr...@google.com <pr...@google.com> #33
The following release(s) address this bug.It is possible this bug has only been partially addressed:
androidx.annotation:annotation-wasm-js:1.9.0-rc01
androidx.annotation:annotation-watchosdevicearm64:1.9.0-rc01
androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.3.2
androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha02
mo...@hungerstation.com <mo...@hungerstation.com> #34
Hello googlers,
I did as this
Our CI stack is Bitrise + Fastlane, I'm getting this error even when i run this sh "gcloud auth activate-service-account --key-file=#{decoded_service_account}"
Execution failed for task ':macrobenchmark:ftlDeviceShiba34Setup'.
A failure occurred while executing com.android.build.gradle.internal.tasks.ManagedDeviceSetupTask$SetupTaskWorkAction Unable to find the application-default credentials to send a request to Firebase TestLab. Please initialize your credentials using gcloud CLI. Examples: gcloud config set project $YOUR_PROJECT_ID gcloud auth application-default login gcloud auth application-default set-quota-project $YOUR_PROJECT_ID Please read
https://cloud.google.com/sdk/gcloud for details.
Description
We are trying to generate Baseline Profiles for our application using the BaselineProfiles plugin and since we want to generate it on our CI/CD pipelines we are exploring Gradle Managed devices.
As there is an already existing issue where Gradle Managed devices fail to setup on Linux machines, we are exploring the new feature of using Firebase Test Lab devices as Gradle Managed devices .
I am following the performance sample app and included the changes mentioned for running tests on Firebase Test Lab devices as Gradle Managed device. When the task
generateBaselineProfile
runs locally, Baseline Profiles are generated correctly. However, when run on the FTL configured device, tests are running successfully, but the profiles are not generated and I see this message in the output:Comparing the logs from local and FTL, I could see that the profiles are generated in a directory different than the task tries to extract from:
When run locally, I can see that it is using the same location from which it is later extracted:
Setup:
Android Studio Hedgehog | 2023.1.1 Canary 5
Build #AI-231.7864.76.2311.10195651, built on May 25, 2023
Runtime version: 17.0.6+0-17.0.6b829.9-10027231 aarch64
AGP 8.2.0-alpha05
Benchmark 1.2.0-alpha13
Could the copying be an issue?
I have attached the performance sample app code with the changes. Run it in the above configuration with FTL setup and should see a similar output.