ElixirConf 2023 - Rafal Studnicki - Keeping real-time auctions running during rollout.

Learn how Whatnot's continuous deployment strategy improved reliability and reduced issues with real-time auctions using blue-green deployment and combined testing.

Key takeaways
  • Real-time auctions require fast and reliable deployment to maintain user experience.
  • Whatnot uses continuous deployment to deliver value to users quickly.
  • During deployment, the auction process can still be running, but it’s not ideal.
  • In this case, a simple change led to a significant improvement in reliability and reduced issues with auctions.
  • Manual deployments were initially ad-hoc and manual, but switching to continuous deployment reduced the risk of issues and allowed for faster testing.
  • The team used a blue-green deployment strategy, where a new version of the service is deployed alongside the old version, and then traffic is routed to the new version.
  • To test the deployment, the team used a combination of synthetic tests and manual testing, and monitored for issues.
  • Problem areas were identified, and the team worked to fix flaky tests and improve deployment reliability.
  • Continuous deployment allowed the team to catch and fix issues quickly, and the impact on user experience was significant.
  • The team uses a variety of tools, including Kubernetes, HORD, and Phoenix, to manage deployments.
  • One challenge the team faced was dealing with auctions that could have different outcomes depending on the version of the service being deployed.
  • The team used a continuous deployment strategy to minimize the risk of issues and ensure that auctions ran smoothly.