Staff Software Engineer - Databases SRE

About the Role

Join Grafana Labs as a Staff Software Engineer to ensure the reliability and scalability of our multi-tenant SaaS databases, Mimir, Loki, and Tempo, which are critical for our highest-SLA customers.

Responsibilities

• Write production code and automation to enhance the resilience, self-healing, and scalability of our core database services and their underlying codebases.
• Lead technical design for new features, ensuring they are scalable and operable from inception.
• Define, evolve, and own the SLO framework for per-tenant reliability models, using data to identify and fix systemic failures.
• Improve observability and alerting systems to proactively detect and address issues before they impact customers.
• Act as a technical escalation point for complex customer environments, leading incident response and writing blameless post-incident reviews.
• Mentor junior engineers and promote reliability best practices across development teams.

Requirements

• 8+ years of software engineering experience, with a strong focus on building and operating distributed systems in production.
• Proficient in at least one object-oriented programming language, with the ability to contribute to large, unfamiliar open-source codebases.
• Deep understanding of distributed systems, including performance, scaling, and failure modes.
• Experience managing multi-tenant systems on Kubernetes within AWS, GCP, or Azure, utilizing infrastructure-as-code tools (e.g., Terraform, Helm).
• Proven experience designing and implementing Service Level Objectives (SLOs).
• Experience with on-call rotations for production systems, including calm troubleshooting under pressure and blameless incident response.
• Demonstrated staff-level technical leadership, including driving cross-team projects and mentoring engineers.
• Working knowledge of Linux internals, networking, and cloud storage.

Nice-to-Haves

• Experience with Mimir, Loki, Tempo, Pyroscope, Prometheus, or other observability/time-series databases.
• Contributions to open-source projects.
• Customer Reliability Engineering (CRE) or customer-facing escalation experience.