Tests Dashboard & Auto-balancing Playwright Shards

[aslushnikov]

Hey folks,

I used to work on Playwright, and I noticed that BlockNote uses Playwright too, including sharding in CI.

These days I’m building Flakiness.io: a GitHub-native test analytics dashboard that tracks flaky tests and detects regressions in pull requests.

We’re prototyping Playwright shard balancing, where Flakiness.io uses historical test timings to produce better-balanced shards. Since BlockNote has a real-world Playwright setup, I’d be very happy if you’re open to testing it on your workload.

I also saw that you already generate a merged Playwright HTML report and upload it as a GitHub artifact. Flakiness.io can simplify that setup: it hosts test reports online for each commit/PR, and adds flakiness, performance, and regression analytics out of the box.

OSS projects can use Flakiness.io for free. I’d be happy to help set it up and see if you find it useful.

[aslushnikov]

FYI: shard balancing is released.

To start using it:

1. Create a project on Flakiness.io to store your test history
2. Install flakiness/playwright
3. Replace npx playwright test with npx flakiness-playwright-shard in Github Actions:

-npx playwright test --shard 1/2 --project foo
+npx flakiness-playwright-shard --shard 1/2 --project foo

It will start balancing shards nicely as soon as it gets initial historic results. Details: https://docs.flakiness.io/shard-balancing/

Let me know if I can help with the integration - we’d love to see how it works for you.