Feature Request: Way to ensure code with no side effects isn't optimised away [199462594]

Bug

Status Update

No update yet.

Description

ma...@google.com

created issue #1

Sep 10, 2021 03:04AM

Hi there, enterprising AOT, Proguard, JIT compilers can sometimes notice that code has no side-effects and can be removed. Even the CPUs are sometimes fast enough to realise that written data can be not committed to memory.

An example might be this code benchmarking String.format:

    @Test
    public void stringFormatWithAllLocales() {
        final BenchmarkState state = benchmarkRule.getState();
        while (state.keepRunning()) {
            for (Locale locale : Locale.getAvailableLocales()) {
                String.format(locale, "%d s", 0);
            }
        }
    }

An enterprising optimiser may 'know' that String.format is side-effect free if the return value isn't used, and optimise it away.

In practice, I don't think this happens for this example. But I've seen a lot of benchmarks where I think they're measuring something, and some compiler has optimised away the code entirely.

Prior art solving this:

allows code to call blackHole.accept(value) and then value will never be optimised away. e.g.:

@Benchmark
public void testMethod(Blackhole blackhole) {
    int a = 1;
    int b = 2;
    int sum = a + b;
    blackhole.consume(sum);
}

Caliper lets you write your Benchmark methods with a return value, then you can accumulate the result of your method call into the return value:

. example:

@Benchmark
long setMaskX64(int reps) {
  long count = 64L * reps;
  long bitMask = 0L;
  for (int i = 0; i < count; i++) {
    bitMask |= 1 << (i & 0x3F);
  }
  return bitMask;
}

Does androidx.benchmark offer anything comparable? Thanks!

Comments

cc...@google.com <cc...@google.com> #2Sep 15, 2021 11:01PM

When I run Macrobenchmark via the `connectedCheck` target with info logs on, I can see the arg being set:

```
INFO: Configuring Android Instrumentation driver: android_instrumentation_runtime {
instrumentation_info {
...
args_map {
key: "androidx.benchmark.suppressErrors"
value: "DEBUGGABLE"
}
}
}
```

Attempted to pass the flag via the `-e` option to the `instrument` command of ADB, but no change to error output:

```
$ adb shell am instrument -w com.ring.benchmark/androidx.test.runner.AndroidJUnitRunner -e androidx.benchmark.suppressErrors DEBUGGABLE
...
ERROR: Benchmark Target is Debuggable
```

ma...@google.com <ma...@google.com> #3Sep 15, 2021 11:12PM

Are you seeing a line like the following? :

ERRORS (not suppressed): ...
WARNINGS (suppressed): ...

Can you try suppressing other errors, does that work? E.g. run the macrobenchmark on an emulator, which should require adding "EMULATOR" to the suppression list, and show it as a suppressed warning.

cc...@google.com <cc...@google.com> #4Oct 13, 2021 01:34AM

I tried this out and it worked for me - I just created a trivial test app and macrobench, test passed. Swapped to debuggable = true in the app, it broke with the expected error, and added the testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE" in the macrobenchmark to make it work again (now with a warning at the top).

Sample project attached.

By the way, note that any measurements you make of your debug app won't be representative of your actual release app.

ma...@google.com <ma...@google.com> #5Oct 13, 2021 01:40AM

Thank you for the fast response. Yes, I see the suppressed messages like:

```
java.lang.AssertionError: ERRORS (not suppressed): DEBUGGABLE
WARNINGS (suppressed):

ERROR: Benchmark Target is Debuggable
Target package com.ringapp.debug is running with debuggable=true in its manifest,
which drastically reduces runtime performance in order to support debugging
features. Run benchmarks with debuggable=false. Debuggable affects execution
speed in ways that mean benchmark improvements might not carry over to a
real user's experience (or even regress release performance).

While you can suppress these errors (turning them into warnings)
PLEASE NOTE THAT EACH SUPPRESSED ERROR COMPROMISES ACCURACY

// Sample suppression, in a benchmark module's build.gradle:
android {
defaultConfig {
testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE"
}
}
at androidx.benchmark.ConfigurationErrorKt.checkAndGetSuppressionState(ConfigurationError.kt:102)
at androidx.benchmark.macro.MacrobenchmarkKt.checkErrors(Macrobenchmark.kt:190)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:229)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:401)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:108)
at com.ring.benchmark.AbstractStartupBenchmark.startup(StartupBenchmark.kt:61)
at com.ring.benchmark.AbstractStartupBenchmark.startupPartialCompilation(StartupBenchmark.kt:51)

```

If I set the EMULATOR flag in addition, and run w/ an emulator attached instead of a physical device:

```
testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "EMULATOR,DEBUGGABLE"
```

I get:

```
java.lang.AssertionError: ERRORS (not suppressed): EMULATOR DEBUGGABLE
WARNINGS (suppressed):

ERROR: Running on Emulator
Benchmark is running on an emulator, which is not representative of
real user devices. Use a physical device to benchmark. Emulator
benchmark improvements might not carry over to a real user's
experience (or even regress real device performance).

ERROR: Benchmark Target is Debuggable
Target package com.ringapp.debug is running with debuggable=true in its manifest,
which drastically reduces runtime performance in order to support debugging
features. Run benchmarks with debuggable=false. Debuggable affects execution
speed in ways that mean benchmark improvements might not carry over to a
real user's experience (or even regress release performance).
```

This leads me to conclude that the gradle declaration is having no effect for some reason. I am guessing that this is not due to Macrobenchmark itself but something in my rather large app build configuration. How can I further debug thigs? For example, I assume that the `testInstrumentationRunnerArguments` declaration causes some state change in the resulting benchmark apk. How can I verify this? I tried unzipping the benchmark apk and seems like a string match in dex files but otherwise no obvious way to see where the issue might be.

cc...@google.com <cc...@google.com> Jan 24, 2024 05:57PM

Reassigned to jg...@google.com.

is...@google.com <is...@google.com> Nov 19, 2024 07:20AM

Status: New

Issue 199462594