User testing in production: how to run a public beta - Eli Holderness - NDC Porto 2023

Run a public beta to validate assumptions, get feedback, and iterate quickly, with tips on communication, documentation, and prioritizing bug fixes and usability issues.

Key takeaways
  • A public beta is a way to run a public experiment to validate assumptions, get feedback, and iterate quickly.
  • It’s essential to have a clear understanding of what you’re trying to achieve with a public beta and to communicate the goals and outcomes clearly.
  • A public beta should be opt-in and reversible, and users should be able to easily exit the beta if they decide it’s not for them.
  • Feedback can come in many forms, including bug reports, usability issues, and feature requests.
  • The most useful feedback often falls into categories such as bugs, usability issues, and new feature requests.
  • Communication is key when running a public beta, especially when introducing changes. Users need to feel heard and acknowledged.
  • Friction can be added to the user journey to encourage people to provide feedback, but it should be balanced with the need for a seamless user experience.
  • Documentation is crucial when running a public beta, as it needs to be clear and concise, and cover both the old and new versions.
  • Public betas can be used in various industries, not just developer tooling, as long as there’s a desire to get feedback and iterate quickly.
  • The goal of a public beta is to get to a minimum viable product (MVP) stage as quickly as possible, and then iterate and refine the product based on user feedback.
  • When running a public beta, it’s essential to prioritize bug fixes and usability issues, and to communicate clearly about what changes are being made and why.
  • A public beta can also help to generate buzz and excitement around a new product or feature, and can be a great way to get users invested in the product’s development.
  • Finally, it’s essential to recognize that public betas are not a guarantee of success, and that there are risks and challenges involved.