IssueTracker

There was an issue overriding coverageFilePath in Android Gradle plugin and Stephan fixed it by https://android.googlesource.com/platform/tools/base/+/ed935d712dd6614bbeaaee2ce539d898c6e6c18d%5E%21/. That fix was cherry-picked into Android Gradle plugin 4.0. Could you try setting the coverageFilePath somewhere outside of your test application package as like the comment#10?

Thank you very much for reporting the issue and waiting for us to address it. We've reinvestigated it and found some more issues. We've fixed it on the head of our development branch. That fix should be included in the next release of the canary version of the Android Gradle plugin, however there will be some limitations.

When you use "clearPackageData" option along with the test coverage enabled, the Android test orchestrator calls "pm clear" to delete all data of your application being tested before each test case. Thus, you have to save your test coverage file in somewhere outside of your application package private storage, '/storage/emulated/0/coverage' for example. It requires you to declare WRITE_EXTERNAL_STORAGE permission in your AndroidManifest.xml file. In addition, you have to grant WRITE_EXTERNAL_STORAGE permission at runtime. Because "pm clear" revokes all the granted permissions, installing the app with "-g" option doesn't help. An easy way to do this is to use GrantPermissionRule.grant(Manifest.permission.WRITE_EXTERNAL_STORAGE), or you can manually grant this permission in your test's teardown if you don't want to grant this permission during the test.

Here's a small working example:

// build.gradle
android {
    defaultConfig {
        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
        testInstrumentationRunnerArguments clearPackageData: 'true'
        testInstrumentationRunnerArguments coverageFilePath: '/storage/emulated/0/coverage'
    }
    buildTypes {
        debug {
            testCoverageEnabled = true
        }
    }
    testOptions {
        execution 'ANDROIDX_TEST_ORCHESTRATOR'
    }
    dependencies {
      androidTestImplementation 'androidx.test:core:1.3.1-alpha02'
      androidTestImplementation 'androidx.test.ext:junit:1.1.3-alpha02'
      androidTestImplementation 'androidx.test:monitor:1.3.1-alpha02'
      androidTestImplementation 'androidx.test:rules:1.3.1-alpha02'
      androidTestImplementation 'androidx.test:runner:1.3.1-alpha02'
      androidTestImplementation 'androidx.test.services:test-services:1.3.1-alpha02'
      androidTestUtil 'androidx.test:orchestrator:1.3.1-alpha02'
    }
}

// ExampleInstrumentedTest.kt
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.rule.GrantPermissionRule

@RunWith(AndroidJUnit4::class)
class ExampleInstrumentedTest {
    @get:Rule
    var mRuntimePermissionRule: GrantPermissionRule =
        GrantPermissionRule.grant(Manifest.permission.WRITE_EXTERNAL_STORAGE)

    @Test
    fun test1() {
    }
}

<!-- AndroidManifest.xml -->
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.orchestratorexample">
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
</manifest>

We're aware of that the requirement of WRITE_EXTERNAL_STORAGE permission in the tested application is not ideal and we're working on the better solution from the test services APK side. I filed a separate bug b/175426813 for tracking it.

Using AGP 7.0.0-alpha14, I'm seeing it put the file in the correct place, however my file is empty and I'm seeing java.lang.IllegalStateException: JaCoCo agent not started, though that may not be related. Currently digging into that, but for those that are saying the testInstrumentationRunnerArguments coverageFilePath: attribute is ignored, that isn't true on the latest alpha AGP.

Alright so the issue I see now with AGP 7.0.0-alpha14 is that the coverage files are stored in an external location on the device, however they're not pulled into the build directory for generating the report, resulting in java.io.IOException: No coverage data to process in directories...

Is there another step I'm missing, or is this a current oversight in the alpha plugin?

I found this stacktrace in logcat

2021-04-16 10:33:38.447 12402-12459/com.example.test.dev E/CoverageListener: Failed to generate Emma/JaCoCo coverage. 
    java.lang.reflect.InvocationTargetException
        at java.lang.reflect.Method.invoke(Native Method)
        at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101)
        at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70)
        at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92)
        at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65)
        at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189)
     Caused by: java.io.FileNotFoundException: /data/data/com.example.test.dev/coverage_data/com.example.test.DepartureListFragmentTest#show_error_no_internet.ec: open failed: ENOENT (No such file or directory)
        at libcore.io.IoBridge.open(IoBridge.java:496)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:235)
        at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
        at java.lang.reflect.Method.invoke(Native Method) 
        at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101) 
        at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70) 
        at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92) 
        at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65) 
        at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392) 
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189) 
     Caused by: android.system.ErrnoException: open failed: ENOENT (No such file or directory)
        at libcore.io.Linux.open(Native Method)
        at libcore.io.ForwardingOs.open(ForwardingOs.java:167)
        at libcore.io.BlockGuardOs.open(BlockGuardOs.java:252)
        at libcore.io.ForwardingOs.open(ForwardingOs.java:167)
        at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:7255)
        at libcore.io.IoBridge.open(IoBridge.java:482)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:235) 
        at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101) 
        at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70) 
        at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92) 
        at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65) 
        at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392) 
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189) 

It seem like CoverageListener.generateCoverageReport run when a new test being executed, at this time app data was cleared by clearPackageData is true. So, no .ec file is copied to external storage.

No, this issue has been fixed already but missing an example project. I'll create an example and add it to https://github.com/android/testing-samples.

Here's my work in progress example project: https://github.com/android/testing-samples/pull/411

By sheer coincidence, I was making my own test sample mostly so that I could make sure that coverage continued to work through future SDK / AGP versions. I've hosted it at https://github.com/growse/Android-Test-Coverage-Sample/ and currently it works when targetting devices up to Android 29.

I'll do some updates so that it makes use of the right dependencies and settings so that it can use the test service and hopefully work on 30+.

You need AGP>7.1 (still in beta), but it does work when using the test services util on devices >=30. I updated my repo to show.

Experiencing the same issues as comment #41 on 7.1.0

2022-01-27T11:25:07.254-0800 [INFO] [com.android.build.gradle.internal.testing.utp.worker.RunUtpWorkAction] 11:25:01 E/Device: Error during Sync: Remote object doesn't exist!

2022-01-27T11:25:07.254-0800 [ERROR] [com.android.build.gradle.internal.testing.utp.worker.RunUtpWorkAction] Jan 27, 2022 11:25:01 AM com.android.tools.utp.plugins.host.additionaltestoutput.AndroidAdditionalTestOutputPlugin deviceShellAndCheckSuccess
WARNING: Shell command failed (1): ls "/sdcard/Android/media/***/additional_test_output"
ls: /sdcard/Android/media/***/additional_test_output: No such file or directory

Jan 27, 2022 11:25:01 AM com.android.tools.utp.plugins.host.additionaltestoutput.AndroidAdditionalTestOutputPlugin afterAll
WARNING: Failed to retrieve additional test outputs from device.
com.android.ddmlib.SyncException: Remote object doesn't exist!
	at com.android.ddmlib.SyncService.pullFile(SyncService.java:341)
        ...

.ec files appear to be generated for each test, but also seem to be unusable -- a report could not be generated from them directly.

Same behavior in 7.1.2. Is there any update on this ticket?

This issue has been fixed. Please refer to this example project: https://github.com/android/testing-samples/blob/main/runner/AndroidTestOrchestratorWithTestCoverageSample/app/build.gradle

It should work in AGP version 7.1.0-alpha12 and above but could you try using the latest AGP version might show you a better warning message? The test coverage data is stored in the additional test outputs directory using the test storage service API. The error message; "WARNING: Failed to retrieve additional test outputs from device. com.android.ddmlib.SyncException: Remote object doesn't exist!", sounds like AGP couldn't create the directory for your testing device.

Please also make sure that you enable "useTestStorageService" and add the dependency in your build.gradle file.

android {
    defaultConfig {
        testInstrumentationRunnerArguments useTestStorageService: 'true'
    }
}

dependencies {
    androidTestUtil 'androidx.test.services:test-services:' + rootProject.testServicesVersion
}

This works https://issuetracker.google.com/issues/123987001#comment44, but it's pretty flaky, approximately three out of 10 runs fail with the below error:

> Task :app:createDebugAndroidTestCoverageReport FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:createDebugAndroidTestCoverageReport'.
> A failure occurred while executing com.android.build.gradle.internal.coverage.JacocoReportTask$JacocoReportWorkerAction
   > Unable to generate Jacoco report

Caused by: java.io.IOException: Unknown block type 6.
        at org.jacoco.core.data.ExecutionDataReader.readBlock(ExecutionDataReader.java:119)
        at org.jacoco.core.data.ExecutionDataReader.read(ExecutionDataReader.java:93)
        at org.jacoco.core.tools.ExecFileLoader.load(ExecFileLoader.java:60)
        at org.jacoco.core.tools.ExecFileLoader.load(ExecFileLoader.java:74)
        at com.android.build.gradle.internal.coverage.JacocoReportTask$JacocoReportWorkerAction$Companion.generateReport(JacocoReportTask.kt:303)
        at com.android.build.gradle.internal.coverage.JacocoReportTask$JacocoReportWorkerAction.execute(JacocoReportTask.kt:269)
        ... 32 more

Please ignore https://issuetracker.google.com/issues/123987001#comment49, fixed it by bumping the below library:

coreVersion = "1.6.0-alpha05"