Lessons from a Hyperscaler • Casey Rosenthal • GOTO 2024

Lessons from a hyperscaler: navigating complex systems with observability and chaos engineering.

Key takeaways
  • Problem with complex systems: it’s hard to understand how they work, and it’s easy to create more complexity.
  • Observability 1.0 was developed by necessity at hyperscalers, but it’s not a solution for everyone.
  • Chaos engineering can help identify weaknesses in complex systems.
  • The four pillars of complexity are: states, relationships, environments, and reversibility.
  • The ability to control these pillars can help navigate complex systems.
  • Observability is a key part of reversibility, allowing for experimentation and the uncovering of systemic weaknesses.
  • Complex systems are characterized by increasing complexity, making it difficult to manage and understand them.
  • The trend is towards more complexity, which can be mitigated by observability and reversibility.
  • Observability is not just about logging, but about providing a single source of truth and allowing for experimentation.
  • Hyperscalers have developed observability tools, but they are not a solution for everyone.
  • Observability is a necessary tool for navigating complex systems, but it’s not a guarantee of success.
  • Complex systems are characterized by unpredictable behavior, making it difficult to make predictions or design systems.
  • Observability can help identify the sources of complexity and allow for better decision-making.
  • The four pillars of complexity are interconnected, and understanding how they interact is key to navigating complex systems.
  • Chaos engineering can help identify weaknesses in complex systems, but it’s not a guarantee of success.
  • Observability is a key part of reversibility, allowing for experimentation and the uncovering of systemic weaknesses.
  • Complex systems are characterized by increasing complexity, making it difficult to manage and understand them.
  • The trend is towards more complexity, which can be mitigated by observability and reversibility.
  • Observability is not just about logging, but about providing a single source of truth and allowing for experimentation.
  • Observability is a necessary tool for navigating complex systems, but it’s not a guarantee of success.
  • The four pillars of complexity are interconnected, and understanding how they interact is key to navigating complex systems.
  • Chaos engineering can help identify weaknesses in complex systems, but it’s not a guarantee of success.
  • Observability is a key part of reversibility, allowing for experimentation and the uncovering of systemic weaknesses.
  • Complex systems are characterized by unpredictable behavior, making it difficult to make predictions or design systems.
  • Observability can help identify the sources of complexity and allow for better decision-making.
  • The four pillars of complexity are interconnected, and understanding how they interact is key to navigating complex systems.
  • Chaos engineering can help identify weaknesses in complex systems, but it’s not a guarantee of success.
  • Observability is a necessary tool for navigating complex systems, but it’s not a guarantee of success.