Data Engineer-Data Platforms-AWS
IBM · Posted Oct 24
Hybrid cloud, artificial intelligence, enterprise software, quantum computing, and IT consulting services
Get a personal compatibility score
Add a resume for personal matches
About the role
IBM Consulting helps clients shape hybrid cloud and AI journeys, with this role focused on building scalable, secure, high-performance data platforms on AWS. The position involves designing data pipelines, serverless and containerized workloads, and large-scale processing with Apache Spark, AWS Glue, Athena, SQL, Python and Scala. The engineer will work with architects, application teams, and business stakeholders to deliver enterprise-grade analytics, reporting, and regulatory data solutions.
What you will do
- Design, develop, and maintain data pipelines and data platforms on AWS
- Build and operate serverless and containerized workloads using AWS Lambda, EC2, and EKS
- Develop batch and large-scale data processing solutions using Apache Spark
- Implement data ingestion, transformation, and analytics using AWS Glue and Athena
- Write complex and high-performance SQL queries for analytics, reporting, and data validation
- Manage and optimize data storage solutions using AWS S3 and RDS
- Design and manage AWS infrastructure using Infrastructure as Code (IaC) with CloudFormation
- Apply AWS security best practices across IAM, VPC networking, data encryption, and access controls
- Develop clean, scalable, and maintainable code using Python and/or Scala
- Monitor, troubleshoot, and optimize performance, cost, and reliability of data workloads
- Collaborate with cross-functional teams to support analytics, reporting, and regulatory use cases
Skills used in this role
What the employer is looking for
- Strong hands-on experience with AWS services, including S3, Lambda, EC2, EKS; AWS Glue, Athena; RDS; VPC, IAM; CloudFormation
- Strong SQL skills, including Writing complex queries, joins, aggregations, and user defined functions; Performance tuning and query optimization
- Good knowledge of AWS Security, including IAM policies, VPC security, encryption, and least-privilege access
- Hands-on programming experience with Python and/or Scala
- Strong experience with Apache Spark for large-scale data processing
- Good communication skills to explain technical concepts to non-technical stakeholders
- Solid understanding of data engineering concepts such as ETL/ELT, data modeling, and distributed systems
Preferred qualifications
- Experience with DBT for data transformations
- Hands-on exposure to Snowflake
- Experience working with Databricks
- Familiarity with modern data lake / Lakehouse architectures
- Experience working in enterprise or regulated environments
- Strong analytical and problem-solving skills
- Ability to work independently and in distributed teams
- Exposure to AWS Toolset: Experience working with AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis to design, build, and operate batch and real-time data pipelines.
- Data Pipeline Development: Exposure to developing batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka.
- Data Layer Creation: Experience working with AWS RedShift, Aurora, and DynamoDB to create data layers and migrate data using AWS DMS.
- Open Source Technologies: Exposure to utilizing open source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on AWS Platform to support data engineering solutions.
- Data Service Management: Experience scheduling and managing data services on the AWS Platform, ensuring efficient operation of data engineering solutions.
- Proficiency with AWS Databrew: Experience working with AWS Glue Databrew to support data engineering solutions, including data preparation and data quality tasks.
- Knowledge of Lambda Functions: Exposure to using Lambda functions with Python to support data engineering solutions, including data processing and data transformation tasks.
- Familiarity with RedShift Spectrum: Experience working with RedShift Spectrum to support data engineering solutions, including data warehousing and data analytics tasks.
Benefits and support
- Compensation and benefits are detailed in the job posting
About IBM
International Business Machines Corporation (IBM) is a global technology and consulting company that provides hybrid cloud, artificial intelligence, and enterprise infrastructure solutions. Operating in over 175 countries, IBM helps businesses across industries modernize operations, secure data, and drive digital transformation through platforms like watsonx and Red Hat OpenShift. Founded over a century ago, the company remains a pioneer in high-performance computing, enterprise automation, and quantum technology.
- Industry
- Information Technology
- Company size
- 260,000+ employees
- Founded
- June 16, 1911
- Location
- Armonk, New York, USA
- Funding stage
- Public Company
Funding
Public Company · $3.5B raised
- ·Post Ipo Debt$150M
Leadership
Chairman, President and Chief Executive Officer
Senior Vice President and Chief Financial Officer
Senior Vice President, Software and Chief Commercial Officer
Senior Vice President, IBM Consulting
Senior Vice President, Marketing and Communications
Recent coverage
IBM Newsroom
Expanding enterprise data engineering with Apache Flink® stream processing2026-10-20
IBM Newsroom
IBM named a Leader in the 2026 IDC MarketScape for Worldwide Agentic AI Platforms2026-10-14
IBM Newsroom
Cleveland Clinic, RIKEN and IBM Finalists for 2026 ACM Gordon Bell Prize in Quantum-Centric Supercomputing2026-09-09
IBM Newsroom
Wimbledon and IBM Introduce New AI-Powered Fan Experiences and Modernized Digital Platforms for The Championships 20262026-07-08