IssueTracker

Thanks for your detailed post.

However, benchmark build type is under configured: at least isProfileable is not set to true for existing build type, probably there's more.

We should set isProfileable = true by default when overriding an existing benchmark build type.

This issue is not about some specific configuration flag, but the general approach of dealing with external configuration. As a developer adopting baseline profiles, it seems extremely risky to me using a custom configuration due to how it's applied under the hood and the fact it may break default configuration.

I agree that is not great but this is a little tricky to do. For custom baseline profile build types we override all the properties. For benchmark I left it open to configure but it's mostly about these 2 properties:

isMinifyEnabled
isShrinkResources

I don't have a way to see if the user is setting them before overriding, so for this reason, I'd prefer not to. I agree with you that some other properties could be set by default to make this easier, i.e.:

isJniDebuggable = false
isDebuggable = false
isProfileable = true

The reason why I mentioned the release signing config in the beginning is because I want to use debug signing config.

In the specific of your issue, i.e. using a debug certificate can you override the benchmark and baseline profile setting? You should be able to do something like:

android {
    buildTypes {
        release { ... }
        debug { ... }
        benchmarkRelease {
            ...
            signingConfig signingConfigs.debug
        }
        nonMinifiedRelease {
            ...
            signingConfig signingConfigs.debug
        }
    }
}

Project: platform/frameworks/support
Branch: androidx-main
Author: Marcello Albano <maralb@google.com>
Link:      https://android-review.googlesource.com/3303459

Added override for debuggable and profileable for benchmark builds in bpgp

Added override for debuggable and profileable for benchmark builds in bpgp

Test: ./gradlew :benchmark:benchmark-baseline-profile-gradle-plugin:test
Bug: 369213505
Relnote: "isProfileable is always overridden in benchmark builds,
  and isDebuggable is also now always overridden in both benchmark and
  nonMinified (baseline profile capture) builds."
Change-Id: I487fa71083921682173f04fcbb477be5baf165f8

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha04

Since release signing config is used by default instead of debug is not mentioned in release notes - maybe this is a bug?

Cause the last time I found it mentioned in the release notes was in version 1.1

signingConfig.debug is used as the default signing config (b/153583269)

So, if the switch to the release one indeed happened - maybe it's an issue?

The new build types should inherit the certificates from the release build type, unless none is specified in which case debug is used: code link

This should be updated in the documentation, thanks for mentioning it. Will take care of updating it.

I am trying to save it in the source after generating by following the DSL specification. This works when the tests run locally on GMD and the profiles are saved in the specified directory as mentioned in the documentation. However, if I switch the GMD to run on FTL, it doesn't save.

I have attached the sample project (Macrobenchmark) and included comments in the build.gradle.kts file in baselineProfile module.

I re-checked the logs related to dex2oat and could see this mismatch warning generated when it's running:

06-01 21:39:39.930: V/installd(463): Running /apex/com.android.art/bin/dex2oat64 in=base.apk out=/data/app/~~Sc5l67aRwmVS1G6tKcZP3w==/com.example.macrobenchmark.target-B8yQDmvRV9qvZZP0CfMurg==/oat/arm64/base.odex
...
06-01 21:39:39.941: W/dex2oat64(9628): Mismatch between dex2oat instruction set features to use (ISA: Arm64 Feature string: a53,crc,-lse,-fp16,-dotprod,-sve) and those from CPP defines (ISA: Arm64 Feature string: -a53,-crc,-lse,-fp16,-dotprod,-sve) for the command line:
06-01 21:39:39.941: W/dex2oat64(9628): /apex/com.android.art/bin/dex2oat64 --zip-fd=7 --zip-location=base.apk --oat-fd=8 --oat-location=/data/app/~~Sc5l67aRwmVS1G6tKcZP3w==/com.example.macrobenchmark.target-B8yQDmvRV9qvZZP0CfMurg==/oat/arm64/base.odex --input-vdex-fd=9 --output-vdex-fd=10 --app-image-fd=13 --image-format=lz4 --profile-file-fd=12 --swap-fd=11 --classpath-dir=/data/app/~~Sc5l67aRwmVS1G6tKcZP3w==/com.example.macrobenchmark.target-B8yQDmvRV9qvZZP0CfMurg== --class-loader-context=PCL[]{} --instruction-set=arm64 --instruction-set-features=default --instruction-set-variant=generic --compiler-filter=speed-profile --compilation-reason=cmdline --max-image-block-size=524288 --resolve-startup-const-strings=true --generate-mini-debug-info --runtime-arg -Xtarget-sdk-version:33 --runtime-arg -Xhidden-api-policy:enabled --runtime-arg -Xms64m --runtime-arg -Xmx512m
...
06-01 21:39:39.963: I/dex2oat64(9628): /apex/com.android.art/bin/dex2oat64 --input-vdex-fd=9 --output-vdex-fd=10 --profile-file-fd=12 --classpath-dir=/data/app/~~Sc5l67aRwmVS1G6tKcZP3w==/com.example.macrobenchmark.target-B8yQDmvRV9qvZZP0CfMurg== --class-loader-context=PCL[]{} --compiler-filter=speed-profile --compilation-reason=cmdline --max-image-block-size=524288 --resolve-startup-const-strings=true --generate-mini-debug-info

which is also present when running on non-ARM devices. I cannot say if this is an ARM specific issue or not, but this issue is prominent when running on ARM devices.

I was looking for something like dex2oat not found. This is what I suspect should happen on the ARM device api 30 where it doesn't work.

In the log you posted here it seems that it's working (although I'm not sure about that mismatch).

Yes, I also couldn't see any such logs. Could you try the sample app attached and check?

Any updates on this one? Will there be a fix in the BaselinesProfile plugin for handling FTL devices and ensuring the DSL works well with them?

Seems like FTL doesn't support this currently but it's on their plan.

I'm not sure about timeline unfortunately.

Thanks for the update. Do you also happen to know if BaselineProfiles plugin will support the results from FTL devices configured as GMD? Second point in my #5 comment mentioned above.

While you can configure FTL devices similarly to GMD, they are fundamentally different and they are managed by different tasks in gradle.

The baseline profile plugin can technically support FTL but it may not work until FTL gradle task support everything that the GMD task do.

Bugjuggler: b/285597357

I am unable to access b/285597357. Do I need any special access?

That task is internal.

This task can be fixed only when that one is completed.

Could the internal issue be made public? It's not clear from the (sparse) docs I could find on managed devices and baseline profile generation that only GMDs were supported.

Thanks for flagging this problem. If you're facing it, use the +1 button.
This issue will be added to the known issues on Baseline Profiles documentation for the time being.

I've come to this issue from the list of known issues of Baseline Profiles plugin. There it says:

Baseline Profile generation isn't supported on Firebase Test Lab devices, including Gradle-managed Test Lab devices

However, comment #5 states that:

When testing the application on a physical device (from FTL) with API 33, the profiles were successfully generated.

So, does it mean that it's actually possible to generate baseline profile using FTL devices?

FWIW, I can generate baseline profiles using FTL ARM device but without AndroidX Baseline Profile Gradle plugin. The FTL Gradle plugin currently doesn't honour instrumentation runner arguments (see b/328216014). As a workaround, I'm passing them via the instrumentation APK manifest.
It would be great if AndroidX Baseline Profile and FTL plugins could finally work properly together, but I don't get my hopes up as the FTL plugin has seen very little development this year.

FWIW, I can generate baseline profiles using FTL ARM device but without AndroidX Benchmark Gradle plugin.

Do you mean w/o androidx.baselineprofile plugin? How do you run the benchmarks/fetch the profiles then? It would be awesome if you could share your setup.

#25 indeed, I meant Baseline profile Gradle plugin. What I did basically is follow this guide to configure FTL in a standalone test module (com.android.test), and tell FTL plugin to pull the contents of the package-specific media directory where the generated baseline profiles are saved.

plugins {
    id("com.google.firebase.testlab")
}

testLab {
    managedDevices{
        create("firebaseSmallPhone") {
            device = "SmallPhone.arm"
            apiLevel = 34
        }
    }

    testOptions {
        results {
            directoriesToPull.set(
                listOf("/sdcard/Android/media/${android.namespace}")
            )
        }
    }
}

At the end of the build, I copy the pulled baseline profile to the src directory:

find <path to baseline profile module>/build/outputs -name '*-baseline-prof.txt' \
  -exec cp {} <path to app module>/src/main/baselineProfiles/baseline-prof.txt \;

https://issuetracker.google.com/issues/285187547#comment26

Thanks!

Could you please specify how you launch a baseline profile generator with this setup? Simply as an instrumentation test?

A little side note. I though that the whole issue is that the folder, where profiles are generated, is always different on an FTL device. With the above setup, what actually prevent you from using the Baseline Profile plugin?

With aosp/3267102, the ftl plugin can now be used with the baseline profile gradle plugin as well. This will be added in version 1.3.2.

In order to use, simply add a baseline profile module, for example :baselineprofile and add the following to build.gradle:

plugins {
    ...
    id("androidx.baselineprofile")
    id("com.google.firebase.testlab")
}

...

firebaseTestLab {

    // Credentials for FTL service
    serviceAccountCredentials.set(file("credentials.json"))

    // Creates one or more managed devices to run the tests on.
    managedDevices {
        "ftlDeviceShiba34" {
            device = "shiba"
            apiLevel = 34
        }
    }

    // Ensures the baseline profile are pulled from the device.
    // Note that this will be automated as well later with aosp/3272935.
    testOptions {
        results {
            directoriesToPull.addAll(\"/storage/emulated/0/Android/media/${android.namespace}\")
        }
    }
}

baselineProfile {
    ...
    managedDevices += "ftlDeviceShiba34"
    useConnectedDevices = false
}

I apologise for posting an offtopic, but attempting to setup Baseline Profiles generation, using the above instructions and FTL Gradle plugin, I've encountered a strange dependency issue: https://issuetracker.google.com/issues/368628050

Firebase section of the issue tracker is not available for me, so posting a reference here with a hope, that somebody with access could check for duplicates and maybe suggest a solution.

Thanks, the dependency issue is a different one already been tracked. Closing this one.

Thanks!

Which version is this fix coming to? Will it be 1.3.2 or we'll need to wait for 1.4.0 release?

The following release(s) address this bug.It is possible this bug has only been partially addressed:

• androidx.annotation:annotation-wasm-js:1.9.0-rc01
• androidx.annotation:annotation-watchosdevicearm64:1.9.0-rc01
• androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.3.2
• androidx.benchmark:benchmark-baseline-profile-gradle-plugin:1.4.0-alpha02

Hello googlers,

I did as this comment #28

Our CI stack is Bitrise + Fastlane, I'm getting this error even when i run this sh "gcloud auth activate-service-account --key-file=#{decoded_service_account}"

Execution failed for task ':macrobenchmark:ftlDeviceShiba34Setup'.

A failure occurred while executing com.android.build.gradle.internal.tasks.ManagedDeviceSetupTask$SetupTaskWorkAction Unable to find the application-default credentials to send a request to Firebase TestLab. Please initialize your credentials using gcloud CLI. Examples: gcloud config set project $YOUR_PROJECT_ID gcloud auth application-default login gcloud auth application-default set-quota-project $YOUR_PROJECT_ID Please read https://cloud.google.com/sdk/gcloud for details.