The art of living with broken things - James Westfall - NDC TechTown 2023

Testing

Discover the art of living with broken things in software testing, exploring the importance of acknowledging failures, embracing trends, and adapting to imperfection to drive true progress and innovation.

Key takeaways

Tests are always failing, even with repeated execution.
The same tests are always passing, despite failures.
Flaky tests aren’t a problem if they’re truly flaky; intermittent failures aren’t a problem if they’re truly intermittent.
End-to-end tests are built for reuse and have an interesting advantage over unit tests.
The art of living with broken things means acknowledging failures and embracing the importance of trends.
True negatives and true positives are essential for understanding test failures.
Overreliance on machine learning can hinder progress; test failures need to be acknowledged and addressed.
Adapting to failure is essential; one cannot improvise their way out of trouble.
The concept of kintsugi, repairing broken things with gold, is relevant to software testing.
Accepting that some things are broken and moving on is crucial.
Trends are more important than instant results; ignoring trends can hinder progress.
End-to-end tests should focus on individual test results, not aggregate results.
Machine learning can be used to classify test failures and automate test execution.
Autotest frameworks can sometimes be a hindrance; manual scripting is often more effective.
Slower and more deliberate testing can be more effective than rapid-fire testing.
Weaknesses in systems are inevitable; acknowledging and addressing them is crucial.
End-to-end tests should prioritize true negatives and actually failing tests over passing tests.
Trends can be obscured by=User {{Current User}}, conflated results and manual analysis can improve trend spotting.

The art of living with broken things - James Westfall - NDC TechTown 2023

More talks