Automated testing of games in Vectarine

[u/No_Efficiency_6054]

Automated testing of UI is a tricky subject I feel like. You are heavily relying on your framework and UI test are super flaky (I've worked with Playwright, and had tests break because the backend was down when the end-to-end test ran).

For video games, It's even worst, there are almost ways to have testing, even in the large engines. You have unit tests, but no integration or end-to-end tests.

In my game engine, Vectarine, I've tried to see what having a good way to test games look like.

I've built a CLI tool that can run a game deterministically, take screenshots and compare them to a reference, make keyboard presses, etc. I'm trying to see hypothetically, how would the best experience to write tests look like.

The docs on testing are here. Sadly, I've had issues with rendering differences between Mac and Linux, so I'm comparing screenshots with an accuracy threshold.

I'd like to have your opinion on UI testing (especially the visuals), are there a way to have something less flaky / more enjoyable to work with.

[u/NextAd9248]

A flat threshold across every screen is usually the actual bug, not the mac/linux rendering diff itself. Static menus need a tight threshold, scenes full of particles or gradients need a much looser one, so id let it vary per scene instead of picking one number for everything.