IssueTracker

Component used:benchmark-macro-junit Version used:1.3.0 Devices/Android versions reproduced on:emulator64_x86_64 REL

We have performance tests that include custom metrics added via TraceSectionMetric. For some reason, I'm seeing a discrepancy between the total number of runs and the number of measurements in the runs array for each metric. This discrepancy occurs only in some runs of the tests, not every time.

My configuration looks like this:

iterations = 20,
startupMode = StartupMode.COLD,
compilationMode = CompilationMode.Partial(warmupIterations = 3),

Here's an example of jq command output:

jq '.benchmarks[] | [{name: .name} + (.metrics | to_entries[] | {(.key): (.value.runs | length)}) + {profilerOutputs: (.profilerOutputs | length)}]' x.y.z-benchmarkData.json

[
  {
    "name": "benchmark12",
    "PapaTtidMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "activityMain.blockingRequestsAccountsFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "activityMain.loginToHomeShowFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "activityMain.mainCreatedToAccountsQueryFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "activitySplash.splashActivityOnCreateFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "activitySplash.splashActivityToPinFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "appStart.applicationOnCreateFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "auth.activityOnCreateFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "auth.authorizationByPinFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "auth.pinScreenVisibleFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "auth.postPreparationFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "disk_mem_testFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "memoryHeapSizeMaxKb": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "memoryRssAnonMaxKb": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "memoryRssFileMaxKb": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "stepAccountsLoadedFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "stepAccountsShownFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "stepAccountsUpdatedFirstMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "timeToFullDisplayMs": 13,
    "profilerOutputs": 20
  },
  {
    "name": "benchmark12",
    "timeToInitialDisplayMs": 13,
    "profilerOutputs": 20
  }
]

I have tried manually requesting some metrics using the Perfetto batch processor and indeed each file contains the requested metric. However, I don't understand why some values are not recorded in the final JSON report. I would prefer to analyze the already generated JSON file.

Any insights into why this might be happening would be greatly appreciated.