When automaticGenerationDuringBuild is enabled and managedDevices are used, benchmark tests fail with:

java.lang.AssertionError: ERRORS (not suppressed): EMULATOR
WARNINGS (suppressed): 

ERROR: Running on Emulator
    Benchmark is running on an emulator, which is not representative of
    real user devices. Use a physical device to benchmark. Emulator
    benchmark improvements might not carry over to a real user's
    experience (or even regress real device performance).

Example configuration in consumer module:

baselineProfile {
    variants {
        // This works
        create("release") {
            automaticGenerationDuringBuild = true
            saveInSrc = true
        }
    }
}

Example configuration in producer module:

baselineProfile {
    useConnectedDevices = false
    managedDevices = [...] 
}