#23 Quantifying Quality - (Pebblebed) Keith Adams | Slush 2022

Quantifying quality is a psychological concept, challenging to measure directly. Insights on managing quality, effective metrics, and the limitations of feedback in achieving "good" experiences.

Key takeaways

Quantifying quality is challenging because it’s a psychological concept that can’t be directly measured.
The only kind of good experience is when nothing bad happens, and all bad things fail to materialize.
Management discipline is necessary to ensure quality.
Using a bug database as an approach to assessing health is not effective.
It’s more useful to focus on the global health of a system, rather than specific metrics.
The lower three charts (performance, quality, and security) don’t align, and need to be treated separately.
The lower one (performance) is most sensitive to changes.
Zeroing in on the sweet spot of “fast enough” is subjective and depends on the system.
A slightly better question to ask is “is this event of sufficient magnitude to tell me something meaningful?”
The masses of customers, who are never going to bother to tell you why they churned, are lost to feedback.
Timely feedback is crucial for learning, and users are unlikely to report problems unless they’re significantly impacted.
Quality is a narrow definition: software behaves as intended.
The numbers game: multiplying slow and 0.99 gets you 0.99, and 0.98 to 0.99 is more significant.
The nines system is a simple way to express quality as a percentage of uptime or responsiveness.
Goodhart’s law: aiming for a target can lead to perverting its meaning.
The alpha and one minus alpha part is an editorial decision about how to weight the two components.
Post-product-market fit, quality gets decoupled from the user’s experience.
Fudging numbers can lead to a pessimistic impression of service quality.
Feedback is often obtained from a biased sample, like customer support tickets, which underreport actual problems.
Directly measuring successes and failures is difficult, but we can do a better-than-chance job of estimating problems.
Measuring quality is a psychological concept, so Goodhart’s law applies. *apter

#23 Quantifying Quality - (Pebblebed) Keith Adams | Slush 2022

More talks