Fighting Test Flakiness with Time Machines - Filip Hric, TestJS Summit 2023

Developers can harness the power of time machines to debug and fix flaky tests, leveraging tools like Replay and Cypress/Playwright to travel back in time, understand what happened, and simplify the debugging process.

Key takeaways
  • Time machines can help fight test flakiness by allowing developers to travel back in time and understand what happened during a flaky test run.
  • Adding automated tests to a flaky app leads to flaky tests, and traditional debugging tools like print statements and debuggers are insufficient for debugging complex scenarios.
  • Replay, a tool that records and replays browser interactions, can be used to create a recording of a test run and then debug the recording.
  • Time machines can be used to learn from the past and apply knowledge gained from debugging to fix flaky tests.
  • The recording can be rewound and replayed to ensure that the issue is reproducible.
  • Manually recreating the issue can be time-consuming and error-prone, while replaying the recording can simplify the process.
  • Replay allows for the creation of snapshots at different points in time, which can be used to analyze what happened during a test run.
  • Time machines can also be used to create a timeline of events, which can help developers understand what happened during a test run.
  • Factors that can contribute to test flakiness include asynchronous operations, complex scenarios, and intermittent issues.
  • Ignoring the app being tested, which is flawed or broken, can lead to flaky tests.
  • Print statements and debuggers are not sufficient for debugging complex scenarios and can lead to “logging chaos”.
  • Cypress and Playwright are tools that can be used to create time machines for debugging purposes.
  • The recording can be shared with others to facilitate collaboration and knowledge sharing.
  • Time machines can be used to reduce the time spent on debugging and fix flaky tests.
  • The concept of time machines can be applied to other areas of software development, such as fixing issues in production environments.