Stabilising eccentric systems - Jessica Brentnall - NDC Oslo 2024

Categorize work into urgency, impact and effort to properly prioritize stabilization efforts:

• High impact/high urgency for security/compliance issues
• High impact/low urgency for technical debt
• Low impact items can go to backlog

Build trust through small experiments and showing measurable results:

• Start with quick wins to demonstrate value
• Share progress and metrics with stakeholders
• Document successes and learnings

Focus on observability fundamentals:

• Review and optimize logging levels/practices
• Clean up noisy alerts and add missing ones
• Implement proper metrics and monitoring

Use appropriate language and communication styles for different audiences:

• Talk about products rather than technical domains
• Adapt messaging for technical vs business stakeholders
• Share regular updates in formats that work for each group

Prioritize automation and process improvements:

• Automate manual deployments and testing
• Document runbooks and procedures
• Streamline incident management processes

Don’t try to fix everything at once:

• Break work into manageable chunks
• Accept that some items can wait
• Focus on highest impact improvements first

Analyze and learn from incidents:

• Do proper post-mortems
• Document root causes
• Implement preventative measures

Build shared understanding of systems:

• Map capabilities and event flows
• Document system interactions
• Create common terminology

Make progress visible:

• Track and share improvements
• Demonstrate reduced incidents/issues
• Show positive impact on team capacity

Consider team health and morale:

• Address pain points affecting developers
• Build in time for improvements
• Celebrate successes and progress