Data Engineer II - Confluent
IBM · Posted Nov 4
Hybrid cloud, artificial intelligence, enterprise software, quantum computing, and IT consulting services
Get a personal compatibility score
Add a resume for personal matches
About the role
IBM Software builds AI-powered, cloud-native products, and the Confluent Data team owns a data platform spanning ingestion, transformation, and data products that support decision-making across the company. The team works with product, engineering, marketing, sales, and field operations stakeholders, using batch and real-time pipelines plus AI-assisted engineering tooling. This Data Engineer II role designs and operates batch and streaming data pipelines, improves platform reliability and tooling, and partners with data scientists and analysts on well-scoped data deliverables.
What you will do
- Design, build, and operate efficient, reliable, and well-documented data pipelines across batch and streaming systems -- from source ingestion through the warehouse to consumer-facing data products.
- Own the data models, SLAs, and quality contracts for the domains you cover; treat documentation, testing, and observability as first-class deliverables.
- Improve the data platform itself: introduce or extend tooling, templates, CI/CD, testing frameworks, and monitoring that make the entire team faster and more reliable.
- Adopt and champion AI-assisted engineering -- use Claude Code and similar tools as part of your daily workflow, build internal AI-powered utilities (text-to-SQL on the semantic layer, automated PR review, log triage, data discovery), and evaluate where LLMs add durable value vs. hype.
- Partner directly with Data Scientists, Analysts, and business stakeholders to translate ambiguous requirements into well-scoped deliverables; communicate trade-offs, anticipate push-back, and drive alignment without needing escalation.
- Mentor interns, new hires, and L2 engineers -- through onboarding, code review, design feedback, and pairing.
- Contribute to cloud infrastructure and governance: IAM, service accounts, secrets management, cost optimization, and Terraform-managed GCP / Confluent Cloud resources.
- Drive incident response and root-cause analysis for the pipelines you own; close the loop with durable fixes, runbooks, and prevention.
Skills used in this role
What the employer is looking for
- 6 to 9 years of experience in Data Engineering at a technology company, including production ownership of non-trivial pipelines and data models.
- Strong command of advanced SQL and Python -- you can write production-grade code, review others' code, and pick the right tool for the job.
- Have end to end ownership of Data pipeline and a clear grasp on requirement gathering, data modeling, data quality and access management
- Solid grounding in dimensional modeling, data warehousing, and ELT/ETL fundamentals; you can design a clean star schema and explain the trade-offs.
- Hands-on experience operating orchestration frameworks in production (Airflow / Cloud Composer, Dagster, or similar).
- Hands-on experience building streaming and event-driven pipelines -- Kafka, Flink, or equivalent -- and reasoning about exactly-once, schema evolution, and back-pressure.
- Experience on a major cloud platform (GCP preferred) and with infrastructure-as-code (Terraform).
- Demonstrated ability to operate with moderate ambiguity: scope your own work, identify what to clarify, recommend an approach, get buy-in, and ship.
- Fluency with AI coding assistants (Claude Code, Cursor, Copilot) as a daily tool -- and judgment about where they help vs. where they hurt.
- Clear written and verbal communication; comfortable presenting designs, post-mortems, and trade-offs to technical and non-technical audiences.
- Experience with open table formats (Apache Iceberg, Delta Lake) and modern lakehouse patterns.
- Familiarity with data quality / observability tooling (Great Expectations, Soda, Elementary, Monte Carlo) and catalog / lineage systems (DataHub, Atlan, Unity Catalog).
- Experience building or integrating with AI/ML infrastructure: feature stores (Feast, Tecton), vector databases (pgvector, Pinecone, Weaviate), or RAG pipelines for internal knowledge systems.
- Experience contributing to or operating a semantic layer that powers self-serve analytics and natural-language interfaces.
- Familiarity with Confluent Cloud, Kafka Connect, and Flink SQL.
- Open-source contributions or technical writing in the data engineering space.
- Bachelor's or Master's degree in Computer Science, Engineering, or equivalent practical experience.
Benefits and support
- Compensation and benefits are detailed in the job posting
About IBM
International Business Machines Corporation (IBM) is a global technology and consulting company that provides hybrid cloud, artificial intelligence, and enterprise infrastructure solutions. Operating in over 175 countries, IBM helps businesses across industries modernize operations, secure data, and drive digital transformation through platforms like watsonx and Red Hat OpenShift. Founded over a century ago, the company remains a pioneer in high-performance computing, enterprise automation, and quantum technology.
- Industry
- Information Technology
- Company size
- 260,000+ employees
- Founded
- June 16, 1911
- Location
- Armonk, New York, USA
- Funding stage
- Public Company
Funding
Public Company · $3.5B raised
- ·Post Ipo Debt$150M
Leadership
Chairman, President and Chief Executive Officer
Senior Vice President and Chief Financial Officer
Senior Vice President, Software and Chief Commercial Officer
Senior Vice President, IBM Consulting
Senior Vice President, Marketing and Communications
Recent coverage
IBM Newsroom
Confluent Introduces Its Streamhouse Architecture to Power AI With Real-Time Business Context2026-11-04
IBM Newsroom
IBM named a Leader in the 2026 Gartner® Magic Quadrant™ for AI Agent Management Platforms2026-11-02
IBM Newsroom
At TechXchange 2026, IBM Announces New Capabilities to Put AI Agents to Work With Governance and Trust2026-10-27
IBM Newsroom
Cleveland Clinic, RIKEN and IBM Team Advance to Finals for 2026 ACM Gordon Bell Prize2026-09-09