Fixed
Status Update
No update yet.
Comments
ap...@google.com
Project: platform/frameworks/support
Branch: androidx-main
commit 91dc18326d0529be713f481918cb1e2537310e6c
Author: Chris Craik <ccraik@google.com>
Date: Tue Sep 14 15:31:20 2021
Add sample-based metric result format
Fixes: 199940612
Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC
Relnote: "Add new type of sampled metric to UI and JSON output, focused on
percentiles of multiple samples per iteration."
Creates a second type of metric result, for percentile-focused metrics
like frame time. These metrics will record multiple values per
iteration, and display distributions (P50,90,95,99) from the entire
collected range of values.
This will drastically simplify the new frame metrics measurements,
both in terms of numbers of values reported in UI, but also in
avoiding showing distribution stats... of distribution stats, as we
did before.
Previous:
frameTime50thPercentileMs min 6.0, median 6.0, max 7.0
frameTime90thPercentileMs min 7.0, median 7.0, max 9.0
frameTime95thPercentileMs min 7.0, median 7.0, max 9.0
frameTime99thPercentileMs min 14.0, median 14.0, max 14.0
totalFrameCount min 294.0, median 294.0, max 294.0
New:
frameTimeMs P50 7.1, P90 9.4, P95 9.6, P99 12.0
Note that this is only used in the FrameTimingTraceMetrics, so isn't
on by default yet.
Similarly, in JSON, we output this data in a new way, as a separate
"sampledMetric" map, e.g.:
"sampledMetrics": {
"frameTimeMs": {
"P50": 7.072010000000001,
"P90": 9.4366065,
"P95": 9.62164825,
"P99": 12.00158825,
"runs": [
[
16.025587,
8.140102,
7.137429,
7.380798,
...
],
...
]
}
}
Change-Id: I56247e421890537d27f1e3f284209aa590472377
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ResultWriterTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/BenchmarkResult.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/MetricResult.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/ResultWriter.kt
M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/api/restricted_current.txt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/IdeSummaryStringTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MetricResultExtensionsTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoResultsParserTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/IdeSummaryString.kt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/IterationResult.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Metric.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MetricResultExtensions.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoResultsParser.kt
https://android-review.googlesource.com/1826075
Branch: androidx-main
commit 91dc18326d0529be713f481918cb1e2537310e6c
Author: Chris Craik <ccraik@google.com>
Date: Tue Sep 14 15:31:20 2021
Add sample-based metric result format
Fixes: 199940612
Test: ./gradlew benchmark:benchmark-common:cC benchmark:benchmark-macro:cC
Relnote: "Add new type of sampled metric to UI and JSON output, focused on
percentiles of multiple samples per iteration."
Creates a second type of metric result, for percentile-focused metrics
like frame time. These metrics will record multiple values per
iteration, and display distributions (P50,90,95,99) from the entire
collected range of values.
This will drastically simplify the new frame metrics measurements,
both in terms of numbers of values reported in UI, but also in
avoiding showing distribution stats... of distribution stats, as we
did before.
Previous:
frameTime50thPercentileMs min 6.0, median 6.0, max 7.0
frameTime90thPercentileMs min 7.0, median 7.0, max 9.0
frameTime95thPercentileMs min 7.0, median 7.0, max 9.0
frameTime99thPercentileMs min 14.0, median 14.0, max 14.0
totalFrameCount min 294.0, median 294.0, max 294.0
New:
frameTimeMs P50 7.1, P90 9.4, P95 9.6, P99 12.0
Note that this is only used in the FrameTimingTraceMetrics, so isn't
on by default yet.
Similarly, in JSON, we output this data in a new way, as a separate
"sampledMetric" map, e.g.:
"sampledMetrics": {
"frameTimeMs": {
"P50": 7.072010000000001,
"P90": 9.4366065,
"P95": 9.62164825,
"P99": 12.00158825,
"runs": [
[
16.025587,
8.140102,
7.137429,
7.380798,
...
],
...
]
}
}
Change-Id: I56247e421890537d27f1e3f284209aa590472377
M benchmark/benchmark-common/src/androidTest/java/androidx/benchmark/ResultWriterTest.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/BenchmarkResult.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/MetricResult.kt
M benchmark/benchmark-common/src/main/java/androidx/benchmark/ResultWriter.kt
M benchmark/benchmark-macro/api/current.txt
M benchmark/benchmark-macro/api/public_plus_experimental_current.txt
M benchmark/benchmark-macro/api/restricted_current.txt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/IdeSummaryStringTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/MetricResultExtensionsTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/StartupTimingMetricTest.kt
M benchmark/benchmark-macro/src/androidTest/java/androidx/benchmark/macro/perfetto/PerfettoResultsParserTest.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/IdeSummaryString.kt
A benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/IterationResult.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Macrobenchmark.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/Metric.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/MetricResultExtensions.kt
M benchmark/benchmark-macro/src/main/java/androidx/benchmark/macro/perfetto/PerfettoResultsParser.kt
Description
Currently, frame times percentiles are treated as individual metrics, with their own internal distributions:
Instead, we should flatmap all metrics captured from separate iterations, and compute metrics from those, e.g:
In order to match this treatment of data in JSON, we should surface percentiles without min/max/median, while still preserving per-iteration separation of data, to enable arbitrary post-processing it.