Back
Rubrik logo

Senior AI Evals Engineer

Rubrik · Posted Oct 15

Cloud data security, zero-trust backup, threat analytics, and cyber resilience services

BangaloreFull-timeOnsiteLead/Staff8+ years₹35.0L–₹55.0L yearly<25 applicants
Cloud ComputingCybersecurityData ManagementSaaSArtificial IntelligencePublic Company
Be an early applicantFull time

Get a personal compatibility score

Add a resume for personal matches

About the role

The Applied AI: Test Engineering team at Rubrik defines, develops, and executes evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) or similar agentic frameworks, with a significant focus on Responsible AI and Safety Evals such as adversarial testing, red-teaming, bias detection, and toxicity measurement. The work covers Rubrik's product verticals — data security, SaaS protection, cyber recovery, and compliance — and bridges AI development with reliable, compliant production deployment. This hands-on Senior AI Evals Test Engineer role (70% automation, 30% manual testing) will design scalable evaluation pipelines, build synthetic test environments and golden datasets, and partner with AI/ML engineers, researchers, and policy teams as a key technical authority.

What you will do

  • Design and build agent evaluation frameworks — create end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion rate
  • Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions, with specific focus on agents built on Google ADK and related Google AI/ML services (e.g., Vertex AI, Gemini models)
  • Build golden datasets and test harnesses — curate domain-specific evaluation sets for Rubrik's product verticals (data security, SaaS protection, cyber recovery, compliance)
  • Establish eval-as-a-service — build and enhance centralized evaluation platform that any IT team members can self-serve to test their agents against standardized and custom benchmarks
  • Automate regression and drift detection — implement CI/CD-integrated eval pipelines that catch regressions on every model or prompt change before production deployment
  • Define eval taxonomies and rubrics — establish structured scoring criteria for agentic workflows (tool use accuracy, multi-step reasoning, hallucination rate, policy adherence, error recovery)
  • Instrument production observability — build real-time monitoring for agent quality in production: success rates, user corrections, fallback triggers, escalation patterns
  • Benchmark and select models — run structured model comparisons (cost, quality, latency trade-offs) to inform model selection decisions across agent use cases
  • Analyze and interpret evaluation results, providing clear, actionable quality and safety reports to stakeholders and development teams, with specific focus on risk metrics, trend analysis, and mitigation recommendations
  • Define comprehensive end-to-end QA strategies — functional, integration, regression, and user acceptance testing (UAT) — tailored for conversational and goal-oriented AI agents with tool-use, memory, and multi-turn reasoning capabilities
  • Develop and maintain detailed test artifacts (test plans, test cases, test scenarios, automated test suites) covering agent features, tool invocation, memory persistence, and reasoning accuracy across evolving model versions
  • Identify, document, prioritize, and track defects, performance regressions, and alignment failures using Jira, collaborating closely with Applied AI Engineers to analyze root causes and validate fixes
  • Evaluate new technologies and patterns (e.g., new tools for Agent Evals), run proofs-of-concept, and influence the platform roadmap
  • Partner with policy and legal teams to ensure agent evaluations systematically cover regulatory compliance, fairness objectives, and emerging AI governance requirements
  • Provide technical leadership to engineers, including solutioning, design reviews, code reviews, pair programming, and mentoring of junior and contract engineers
  • Collaborate cross-functionally — work closely with ML engineers, product teams, platform engineers, and security to embed eval-driven quality gates into the agent development lifecycle

Skills used in this role

PythonTypeScriptJavaSeleniumPlaywrightpytestLangSmithDeepEvalPromptfooRAGASHugging FaceLangGraphClaude Agent SDKGoogle Agent Development KitVertex AIGeminiAnthropic ClaudeOpenAIAzure OpenAIGoogle Cloud PlatformKubernetesJenkinsGitHub ActionsDatadogGrafana

What the employer is looking for

  • Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
  • 8+ years of overall software Test engineering experience, with 5+ years focused on test engineering, QA automation, or developer tooling, and demonstrated ability to build evaluation systems across multiple AI/ML domains, conversational agents, or Large Language Models (LLMs)
  • 2+ years working with ML/AI systems, building, testing, or evaluating models in production environments, including hands-on experience with LLM-based applications, prompt engineering, RAG pipelines, or agentic workflows
  • 2+ years as a senior/lead, owning evaluation strategy, quality frameworks, and technical decisions for complex AI agent systems in an enterprise product environment
  • 1+ years in designing or operating evaluation frameworks specifically for generative AI / LLM agents, including benchmarking, red-teaming, and production quality monitoring in the AI ADLC model
  • Deep hands-on experience with LLM evaluation — familiarity with frameworks and libraries such as LangSmith, DeepEval, LLM as a Judge, Promptfoo, RAGAS, Hugging Face evaluation tooling, custom eval harnesses
  • Strong understanding of agentic AI patterns — tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, retrieval-augmented generation (RAG)
  • Production experience with eval pipelines — building automated, CI-integrated evaluation infrastructure (not just notebooks)
  • Proficiency in Python — strong scripting and tooling skills; experience with pytest, async patterns, and data processing at scale
  • Familiarity with Google Cloud Platform (GCP) or any Cloud Service relevant to AI/ML (e.g., Vertex AI) and experience integrating testing into MLOps workflows
  • Statistical literacy — ability to design experiments, interpret eval metrics with appropriate confidence, and avoid common pitfalls (overfitting to benchmarks, metric gaming)
  • Enterprise product sensibility — understanding of how AI quality translates to customer trust in a B2B/enterprise context (security, compliance, reliability SLAs)
  • Excellent communication — ability to present eval results and quality trade-offs to engineering leadership, articulate risks and safety findings to non-technical audiences, and drive alignment on ship/no-ship decisions
  • Hands-on experience with CI/CD pipelines and Git-based workflows, including automated testing, static analysis, deployment automation, and environment configuration for our DevTestOps
  • Languages: Python(Must), TypeScript, Java (Optional)
  • Test Automation: Selenium, Playwright
  • LLM Providers: Anthropic Claude, OpenAI, Azure OpenAI, open-source models
  • Eval Tooling: DeepEvals(Must), LLM as a Judge (Must), Promptfoo/RAGAS/LangSmith/ Custom harnesses
  • Agent Evals Frameworks/libraries: LangGraph, Claude Agent SDK, custom orchestrators
  • Infrastructure: DevOps (Basic), Kubernetes, CI/CD (Jenkins/GitHub Actions), observability stacks (Datadog/Grafana)
  • Proven ability to lead without direct authority, influencing cross-functional teams and driving technical decisions through clarity and data rather than hierarchy.
  • Experience working in agile environments, collaborating closely with product owners, scrum teams, and distributed stakeholders across time zones.
  • Excellent communication and documentation skills, comfortable presenting solutions and trade-offs to engineers, managers, and senior business stakeholders.
  • Strong architectural thinking and problem-solving skills, including the ability to decompose complex business processes into simple solutions

Preferred qualifications

  • Deep Learning/NLP Knowledge: Fundamental understanding of how LLMs work, including concepts like RAG, prompting, and the technical challenges related to alignment
  • Data Analysis: Experience with data analysis and visualization tools (e.g., Pandas, SQL) for analyzing evaluation and safety results
  • Policy Background: Familiarity with AI ethics principles and industry standards for responsible AI development
  • Garbage-in/Garbage-out (GIGO) Tools: Any tool or framework that specializes in testing the model's response to highly ambiguous, misleading, or malformed inputs to test for robustness and safety failures
  • Experience testing complex APIs, especially for tool-integration and function-calling scenarios in agentic workflows
  • Experience with prompt injection (Must) and data poisoning skills is a plus
  • Certifications in AI/ML, cloud platforms (GCP, AWS), AWS Machine Learning Specialty, Google Professional ML Engineer, DeepLearning.AI specializations and software testing methodologies
  • Experience evaluating agents in cybersecurity, data protection, or infrastructure domains

Benefits and support

  • Compensation and benefits are detailed in the job posting

About Rubrik

Rubrik logo

Rubrik

Company website
LinkedInXCrunchbaseGlassdoor★4.0

Rubrik is a cloud-native cybersecurity and data resilience company that provides zero-trust data protection, threat monitoring, and automated enterprise recovery solutions. The company delivers a unified platform designed to safeguard hybrid and multi-cloud environments against ransomware and malicious cyber threats. By integrating AI-driven operations and risk analytics, Rubrik helps enterprises ensure business continuity and digital trust.

Industry
Cloud Computing
Company size
3000-5000 employees
Founded
2014
Location
Palo Alto, California, USA
Funding stage
Public Company

Funding

Public Company · $553M raised

Greylock PartnersLightspeed Venture PartnersMicrosoftKhosla VenturesInstitutional Venture Partners (IVP)
  • 2021-08-17Series FUndisclosed
  • 2021-11-01Secondary Market$1.3M

Leadership

BS
Bipul Sinha

Co-Founder, CEO and Chairman

AN
Arvind Nithrakashyap

Co-Founder and CTO

KC
Kiran Choudary

Chief Financial Officer

AG
Anneka Gupta

Chief Product Officer

Recent coverage

Apply on company site