Senior Software Engineer, GenAI & ML Evaluation Frameworks

About the Role

Join Grafana Labs, an open-source powerhouse, to play a key role in shaping how users understand complex observability data through AI-driven features. This exciting opportunity focuses on building and evolving internal evaluation frameworks for Generative AI and Large Language Models (LLMs) to ensure high-quality, reliable AI capabilities.

Responsibilities

• Design and implement robust evaluation frameworks for GenAI and LLM-based systems, including golden test sets, regression tracking, LLM-as-judge methods, and structured output verification.
• Develop tooling to enable automated, low-friction evaluation of model outputs, prompts, and agent behaviors.
• Define and refine metrics for both structure and semantics, ensuring alignment with realistic use cases and operational constraints.
• Lead the development of dataset management processes and guide teams across Grafana in best practices for GenAI evaluation.

Requirements

• Proven experience designing and implementing evaluation frameworks for AI/ML systems.
• Familiarity with prompt engineering, structured output evaluation, and context-window management in LLM systems.
• High autonomy to collaborate effectively and translate team goals into clear, testable criteria supported by effective tooling.

Nice-to-Haves

• Experience working in environments characterized by rapid iteration and experimental development.
• A pragmatic mindset that values reproducibility, developer experience, and thoughtful trade-offs when scaling GenAI systems.
• Passion for minimizing human toil and building AI systems that actively support engineers.