Status Update
No update yet.
Comments
ma...@gmail.com
I tried adding screenshotTestImplementation(libs.androidx.window) but it didn't fix it.
ba...@gocity.com
This is likely a duplicate of libs.androidx.window dependency to implementation to see if it works?
dm...@osinit.com
Any updates on this issue?
ma...@gmail.com
please refer to this investigation done on paparazi for potential solutions and where the issue probably stems from (with giving credit to the jrodbx author for the investigation and proposed solution of course)
ja...@google.com
The difference between platform is always going to happen, as the screenshots are taken on simulated environments that use platform calls at the end of the chain to render text, shadows, etc. This issue has been documented here:
This is unrelated to this issue, but BOM 2024.10.01 introduced a change in elevation and shadows that might break a lot of screenshots. We are discussing how to give a heads-up next time this happens.
ma...@gmail.com
For a 1000x1000px box where pixels differ by just one off (0xFF003EB0 vs 0xFF003EB1), what's actually a 'sane' threshold? 0.01%? 0.1%? 0.4%? Does it depend on image size or average difference? Can you provide better insight there?
The suggestion to generate golden images on CI fundamentally breaks developer workflow and creates fragile tests. Every CI machine OS or library update would potentially invalidate ALL golden images, creating a maintenance nightmare.
Developers need to generate and verify UI tests locally during development, not push changes just to get reference images from CI. A screenshot testing tool should adapt to developer workflows, not force developers to adapt around tool limitations.
The fact that Paparazzi analyzed this and achieved matching across platforms shows this is (mostly) solvable at the diffing/rendering level - yes, it's a hard problem, but it's not impossible. Instead of working around rendering inconsistencies with CI constraints and arbitrary thresholds, efforts should focus on making the rendering or diffing pipeline produce consistent results across environments.
ja...@google.com
I agree, the goal should be consistent results across environments.
However, both Robolectric and LayoutLib teams have confirmed that it would be an gargantuan task. And I'll add: probably full with bugs in the beginning.
The suggestion to generate golden images on CI fundamentally breaks developer workflow and creates fragile tests.
Agree with changing the developer workflow, but why would it create fragile tests? On the contrary.
If you want to continue running tests locally, you can simply record new tests before starting to work on the change. You can also omit that and run the validation tasks, expecting the small differences. Most of the time I just let CI take care of it.
ma...@gmail.com
a developer in a large team is doing a small change, creates Previews for screenshot tests, pushes, CI generates "golden" images
- if there was a lib or similar update that would do a major change in how rendering is done, all the screenshots would be regenerated even for unrelated changes(?), and would be added in a commit (at least for the flow mentioned in docs you sent)
- or validation without any new screenshots for that sense would also fail, no need to generate new images, update the OS, drivers, libs etc. old golden images will not be valid anymore when cross validating them in regular CI flows.
either way we will observe how this library proceeds on this problem and if it gets resolved, in case it remains potentially flaky we will consider how to adapt to the potential issues or consider alternatives. thanks!
Description
Updating to alpha02 produced slightly different colors in testing-samples:https://github.com/android/testing-samples/pull/514
This only happens on CI, I can't reproduce locally for now. Using linux on both platforms.
TODO: Figure out which of the images is correct.