
Staff Software Engineer, Frontier Security Team
Snowflake · Posted Sep 25
Cloud data warehousing, analytics, data sharing, and AI infrastructure services
Get a personal compatibility score
Add a resume for personal matches
About the role
Snowflake's Frontier Security AI teams develop production-grade LLM applications, intelligent agents, AI infrastructure, and evaluation systems for enterprise customers — products that must meet a high bar for quality, security, reliability, and efficiency while operating over sensitive data at large scale. In this role, you will lead the design and development of the Agentic Harness and agent evaluation platform, working across product, infrastructure, applied AI, security, and modeling teams to take new capabilities from prototype to dependable customer value.
What you will do
- Architect and build the Agentic Harness that executes complex, multi-step AI workflows across models, tools, data, and services.
- Design stable interfaces for tool execution, context construction, state management, memory, permissions, retries, fallbacks, and human review.
- Own agent quality end to end by building evaluation harnesses, representative datasets, automated graders, experiment pipelines, and release gates.
- Convert ambiguous reports such as "the agent feels worse" into measurable failure modes, reproducible tests, and durable fixes.
- Analyze production agent trajectories to identify failures in reasoning, retrieval, tool use, context, orchestration, and application code.
- Close the loop between production incidents, root-cause analysis, evaluation coverage, and regression prevention.
- Develop offline and online measurements for task completion, correctness, groundedness, safety, latency, reliability, and cost.
- Build simulation and replay infrastructure for golden-set tests, adversarial scenarios, model comparisons, and large-scale experiments.
- Improve agent efficiency through model routing, prompt and semantic caching, context compaction, tool-result management, and token optimization.
- Productionize new model capabilities as secure, observable, multi-tenant services with clear operational controls.
- Establish standards for evaluation design, including sampling, ground-truth quality, grader calibration, leakage prevention, and statistical significance.
- Define technical direction across multiple teams and lead projects whose scope extends beyond a single service.
- Mentor engineers, raise the quality of architecture reviews, and remain directly involved in implementation and debugging.
Skills used in this role
What the employer is looking for
- 9+ years of software engineering experience, including technical leadership of complex production systems.
- Direct experience shipping and operating LLM applications, AI agents, or model-backed workflows in production.
- Strong background in distributed systems, service architecture, high-throughput APIs, concurrency, and failure handling.
- Experience building an agent runtime, workflow engine, developer platform, evaluation system, or similar infrastructure.
- Demonstrated ability to evaluate nondeterministic systems without relying on a single aggregate score.
- Fluency in Python and strong proficiency in at least one systems or application language such as Java, Go, Rust, or TypeScript.
- Hands-on knowledge of tool calling, structured generation, retrieval, context engineering, prompt management, and model APIs.
- Experience with production observability, including structured traces, replay, metrics, logs, and incident diagnosis.
- Ability to balance agent quality with latency, reliability, security, and inference cost.
- Track record of setting technical direction and delivering results across organizational boundaries.
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
- Clear written and verbal communication with engineering, product, and leadership audiences.
Preferred qualifications
- Building evaluation or observability infrastructure for agentic coding, data engineering, or analytics systems.
- Designing human-evaluation programs, scoring rubrics, annotation workflows, or grader-calibration methods.
- Working with multi-agent orchestration, long-running agents, asynchronous workflows, or durable execution.
- Developing synthetic tasks, simulations, adversarial tests, red-team exercises, or safety guardrails.
- Building retrieval systems that use vector search, hybrid search, semantic indexing, ranking, or caching.
- Operating multi-tenant systems that process sensitive enterprise data.
- Working with model training, fine-tuning, reinforcement learning, or feedback-driven optimization.
- Evaluating and onboarding frontier models based on measured product outcomes.
- Experience with databases, SQL engines, data platforms, Kubernetes, or cloud-native infrastructure.
- Treat evaluation as part of product engineering rather than a final validation step.
- Can move between agent behavior, distributed infrastructure, data analysis, and production debugging.
- Question metrics that do not reconcile and design tests that can expose misleading results.
- Take ownership from early architecture through deployment, operations, and measurable customer outcomes.
- Prefer evidence from representative tasks and production data over intuition alone.
Benefits and support
- Compensation and benefits are detailed in the job posting
About Snowflake
Snowflake delivers the AI Data Cloud, a cloud-based platform that unifies data warehousing, data lakes, data engineering, and data sharing. Its architecture separates compute from storage, enabling organizations to securely store, govern, and analyze data across multiple public clouds. The platform also integrates generative AI services and developer frameworks to help enterprises scale advanced workloads.
- Industry
- Cloud Computing
- Company size
- 10001+ employees
- Founded
- 2012-07-23
- Location
- Bozeman, Montana, USA
- Funding stage
- Public Company
Funding
Public Company · $1.56B raised
- 2012-02-01Seed$900K
- 2012-08-01Series A$5M
- 2014-10-01Series B$26M
- 2015-06-01Series C$79M
- 2017-09-01Series D$105M
- 2018-01-01Series E$263M
- 2018-10-01Series F$450M
- 2020-02-01Series G$479M
- 2020-09-16Post IPO$376M
Leadership
Chief Executive Officer
Chief Financial Officer
Executive Vice President, Product
Chief Technology Officer
Chief Security and Trust Officer
Recent coverage
Snowflake Press Release
Snowflake to Showcase Enterprise-Wide Agentic Transformation at EXPEDITION 20262026-10-05
Snowflake Press Release
LSEG and Snowflake expand collaboration to connect trusted financial data with cloud-native AI workflows2026-09-30
Snowflake Press Release
Snowflake Prices Upsized Private Placement of $3.75 Billion of 0.00% Convertible Senior Notes2026-09-29
Snowflake Press Release
Snowflake Unlocks Better AI Economics with Dynamic Model Routing, Delivering More Value to Customers2026-08-18