Assigned
Status Update
No update yet.
Comments
cc...@google.com
Can you call the callback yourself? If it needs to be on the UI thread, you can use @UiThreadTest, just as with any instrumentation test.
Generally, I'm reluctant to add more complex APIs for scoping measurement:
1) Multithreaded benchmarks are guaranteed to be less stable. The kernel gives its most stable performance when threads run continually without sleeping.
2) It's generally inefficient to measure one small part of a large chunk of code - ideally, just build whatever the input is once, capture it, and run/measure the next stage after that in the loop. This gives you ideal performance, and isolation of what you're measuring from other code running inside the loop.
3) other platforms that the benchmarks might run on (e.g. host JVM) won't necessarily support new APIs I add to androidx benchmark, so if compose wants to eventually benchmark on host machines as well, we may have to build a wrapper layer ourselves that adds the same functionality to other benchmark libraries for host
4) If the timing isn't at the loop boundary, it increases interference, since we can't measure once across many loop iterations for small benchmarks, and it's difficult to factor out measurement overhead otherwise. As we add more and more metrics to androidx benchmark, we'll keep increasing the interference for every benchmark with any custom timing (including `runWithTimingDisabled`)
Generally, I'm reluctant to add more complex APIs for scoping measurement:
1) Multithreaded benchmarks are guaranteed to be less stable. The kernel gives its most stable performance when threads run continually without sleeping.
2) It's generally inefficient to measure one small part of a large chunk of code - ideally, just build whatever the input is once, capture it, and run/measure the next stage after that in the loop. This gives you ideal performance, and isolation of what you're measuring from other code running inside the loop.
3) other platforms that the benchmarks might run on (e.g. host JVM) won't necessarily support new APIs I add to androidx benchmark, so if compose wants to eventually benchmark on host machines as well, we may have to build a wrapper layer ourselves that adds the same functionality to other benchmark libraries for host
4) If the timing isn't at the loop boundary, it increases interference, since we can't measure once across many loop iterations for small benchmarks, and it's difficult to factor out measurement overhead otherwise. As we add more and more metrics to androidx benchmark, we'll keep increasing the interference for every benchmark with any custom timing (including `runWithTimingDisabled`)
mo...@google.com
I'll try to rearrange my tests.
I'm not sure that the API is more complex.
I'm not asking for a multithreaded benchmark. I'm asking for a single-threaded benchmark in which I can select to "start" and "stop" the timing without using a scoped timing/non-timing. Does that help?
I'm not sure that the API is more complex.
I'm not asking for a multithreaded benchmark. I'm asking for a single-threaded benchmark in which I can select to "start" and "stop" the timing without using a scoped timing/non-timing. Does that help?
Description
In any case, I can't easily do "measureRepeated" for such a case. I have setup CountDownLatches to mark the start and end of the time and done this:
measureRepeated {
runWithTimingDisabled {
startLatch = CountDownLatch(1)
endLatch = CountDownLatch(1)
pokeCallbackToHappen()
startLatch.await() // startLatch.countDown() is called at the beginning
}
endLatch.await() // endLatch.countDown() is called at the end of callback
}
That's not ideal, but it works. I'd prefer having some way to mark the start time and end time so that I don't have to use the wrong thread to wait for timing.