IssueTracker

Ideally there would be two thresholds we could customize:

1. Pixel difference threshold, i.e how much any given pixel can change before it's considered different from the target pixel.
2. Total difference threshold, i.e how much of the total screenshot can be different from the target

In my situation, the ideal settings (and what I currently use on other tools/platforms) is ~1% pixel difference threshold to ignore text rendering and anti-aliasing differences that occur across hardware and a 0% total difference threshold so that any change more significant than that is flagged immediately.

Using a single total difference threshold to ignore aliasing is unideal, because on screenshots with text this value can end up being very large to account for every possible pixel that could receive different aliasing. So large that other actually important differences may sneak by undetected, e.g. the height or color of a divider line.

I like the idea of having multiple thresholds. Could you please explain option 2 (total diff) in more detail?

Option 2 (total difference) is the percentage of the image that is allowed to be 'different' from the reference image. For example, if the reference image is 10x10 pixels, a 5% threshold would allow the test to pass even if up to 5 pixels do not match the reference. In my opinion, this is the least useful threshold option, even though most other screenshot testing tools have it, as it's the most straightforward threshold to implement. I avoid using this kind of threshold because it doesn't scale well with large screenshots. If I'm taking a screenshot of an entire screen of my app, say 1080x1920, even a small threshold of 0.5% allows for over 10,000 pixels to not match the reference. This means that it's possible the test won't catch important changes like the color of a border, a change of font, or potentially even a small icon disappearing.

What I find to be a more useful option is configuring when any given pixel is considered different from the reference pixel. For example, if I have a screenshot of an black circle on a white background, my laptop (an Apple Silicon MacBook) will anti-alias the image slightly differently than a colleague's Windows desktop or the Linux instance running my tests for CI. Even though to the human eye the image is identical, it will fail due to the minor differences in the gray pixels used to anti-alias the edge of the circle. In this situation, I don't care if a pixel that was #9A9A9A is now #9B9B9B; they're essentially identical, and I'd want the test to ignore that difference.

Trying to work around this kind of issue with option 2 isn't ideal. Larger screenshots have more anti-aliasing, which necessitates a larger total difference threshold to ignore the larger number of imperceptible differences, which in turn just increases the potential for a meaningful change to not be detected.

seems someone is working on this check out this commit. Tried it in alpha-05 but didn't work so I suspect it will be available in alpha-06.

For configuring check the ScreenShotTest.kt file

Diff threshold feature is enabled since alpha06 version. This does not solve the platform related rendering differences. Tracking that as a separate issue and closing this