Senior AI Product Ops
Mastercard · Posted Oct 7
Global technology company connecting consumers, financial institutions, merchants, and businesses through secure payment processing and digital commerce solutions
Get a personal compatibility score
Add a resume for personal matches
About the role
Mastercard's Operational Intelligence team is building the next generation of data products and AI solutions that help customers access and use operational insights in new ways. As AI agents move from POC to production, their behavior keeps changing in real time, so their reliability, safety, and trustworthiness must be actively operated. This AI Ops role owns end-to-end production monitoring, evaluation, triage, and governance for one or more AI agents in partnership with AI Engineering.
What you will do
- Own end to end production monitoring for one or more AI agents, tracking key metrics, KPIs, accuracy, latency, SLA breaches, guardrail triggers, and out of scope rates through dedicated observability dashboards
- Serve as L1/L2 production support, providing first response monitoring, triage, and escalation to tech teams as new agentic services go live
- Identify and diagnose issues by monitoring eval scores and drift alerts, inspecting failing traces and patterns, and classifying root causes such as intent, tool, parameter, or hallucination errors
- Evaluate whether the agent’s reasoning remains correct in production, checking whether context retrieval stays current, whether resolutions match ground truth, and whether the learning loop is drifting
- Monitor and manage post production reliability, including API and authentication failures and issues originating from external AI endpoints, to protect revenue generating, client facing agents
- Partner with AI Engineering to prioritize fixes, run experiments on prompts, context, and routing in Dev, and validate improvements via evals before handing off proven changes for deployment
- Embed governance within the agent’s learning loop so that approval workflows and audit logging travel with every production change, rather than being added after the fact
Skills used in this role
What the employer is looking for
- Experience monitoring or operating production AI/ML or agentic systems, including observability, evaluation, and drift detection practices
- Strong analytical skills to diagnose root causes across intent classification, tool selection, parameter errors, and hallucination patterns
- Understanding of AI/LLM observability concepts such as tracing, telemetry, guardrails, and evaluation engineering including benchmarks, automated evals, human review, and regression testing
- Familiarity with governance, compliance, and responsible AI practices such as PII detection, access control, audit logging, and policy enforcement is a plus
- Ability to work independently alongside existing Tech teams while owning a dedicated agent’s post launch health
- Excellent communication skills to translate production signals into clear, prioritized asks for AI Product and AI Engineering teams
- Comfort working in a fast evolving environment where the process itself, not just the throughput, is what is being continuously operated on and improved
Benefits and support
- Compensation and benefits are detailed in the job posting
About Mastercard
Mastercard is a global technology company in the payments industry that connects consumers, financial institutions, merchants, governments, and businesses worldwide. It operates a fast and secure payment processing network that spans more than 200 countries and territories. Through innovative products and solutions in credit, debit, digital assets, and security, Mastercard makes commerce safer, simpler, and more accessible.
- Industry
- Payments
- Company size
- 10001+ employees
- Founded
- 1966
- Location
- Purchase, New York, USA
- Funding stage
- Public Company
Funding
Public Company · $115M raised
Leadership
Chief Executive Officer
Chief Business Officer
President & Chief Technology Officer
Chief People Officer
Recent coverage
Mastercard Newsroom
Mastercard advances agentic commerce with new trust and intelligence servicesSeptember 30, 2026
Mastercard Newsroom
Mastercard launches AI-powered B2B analytics for supplier paymentsSeptember 30, 2026
Mastercard Newsroom
Mastercard completes acquisition of BVNK to advance global stablecoin capabilitiesAugust 3, 2026
Mastercard Newsroom
Mastercard launches Wallet Pay to scale digital wallets worldwideSeptember 10, 2026