Lead Software Engineer - Pyspark, AWS, Python
JPMorgan Chase · Posted Oct 5
Global banking, investment, credit card, mortgage, and asset management services
Get a personal compatibility score
Add a resume for personal matches
About the role
JPMorgan Chase's Consumer and Community Banking division runs an agile team that enhances, builds, and delivers trusted market-leading technology products in a secure, stable, and scalable way. The team works across cloud-based data engineering, large-scale data processing, and analytics platforms supporting the firm's business objectives. As a Lead Software Engineer and core technical contributor, this role leads the design of scalable cloud data pipelines, architects data models for large-scale datasets, defines data strategy, and drives data quality and governance across multiple technical areas.
What you will do
- Lead the design, development, and maintenance of robust, scalable cloud-based data processing pipelines and infrastructure, ensuring adherence to engineering standards, governance frameworks, and industry best practices.
- Architect and refine data models for large-scale datasets, optimizing for efficient storage, high-performance retrieval, and advanced analytics while upholding data integrity and quality.
- Partner with cross-functional teams to translate complex business requirements into effective, scalable data engineering solutions that drive organizational value.
- Champion a culture of innovation and continuous improvement, proactively identifying and implementing enhancements to data infrastructure, processing workflows, and analytics capabilities.
- Define and execute data strategy, including the development of enterprise data models and the management of end-to-end data infrastructure—from design and construction to installation and ongoing maintenance of large-scale processing systems.
- Drive data quality initiatives, ensure seamless data accessibility for analysts and data scientists, and maintain strict compliance with data governance and regulatory requirements.
- Align data engineering practices with business objectives, ensuring solutions are both technically sound and strategically relevant.
- Author, review, and approve technical requirements and architectural designs, and lead process re-engineering efforts to deliver cost-effective, high-impact business solution
- Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.
- Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.
Skills used in this role
What the employer is looking for
- Formal training or certification on software engineering concepts and 5+ years applied experience
- Expert in at least one distributed data processing framework (Spark). Expert in at least one cloud data Lakehouse platforms (AWS Data lake services or Databricks, if not Hadoop),
- Expert in at least one scheduling/orchestration tools ( Airflow, alternatively AWS Step Functions or similar) & Expert with relational and NoSQL databases. Expert in data structures, data serialization formats (JSON, AVRO, Protobuf, or similar), and big-data storage formats (Parquet, Iceberg, or similar)
- Hands-on professional experience in one or more programming language(s), including Java or Python, proficiency in Python, SQL, and at least one additional language (e.g. Java or Scala) for data engineering tasks
- Hands-on experience utilizing Apache Spark for large-scale data processing, including developing and optimizing data pipelines, performing real-time and batch analytics, and leveraging Spark’s libraries for machine learning and data transformation to drive actionable business insights.
- Proficiency in microservices architecture, serverless computing and distributed cluster computing tools such as Docker, Kubernetes etc. Experience in one or more data modelling techniques (Dimensional, Data Vault, Kimball, Inmon, etc.)
- Experience with test-driven development (TDD) or behavior-driven development (BDD) practices, as well as working with continuous integration and continuous deployment (CI/CD) tools.
- Experience organizing and leading design workshops, coding sessions, and hackathons to promote a culture of excellence and innovation in data engineering. Expertise in architecting reusable, future-ready design patterns that address diverse use cases across the organization.
- Expertise in working with streaming platforms like Kafka, MQ etc.
- Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
- Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices
Preferred qualifications
- Hands-on experience with Infrastructure as Code (IaC) tools, preferably Terraform; experience with AWS CloudFormation is also valued.
- Proficiency in cloud-based data pipeline technologies such as Spinnaker or similar platforms.
- Strong working knowledge of the Snowflake data platform.
- Experience in budgeting and resource allocation for data engineering projects.
- Proven ability to manage vendor relationships effectively.
Benefits and support
- Compensation and benefits are detailed in the job posting
About JPMorgan Chase
JPMorgan Chase & Co. is one of the oldest, largest, and most diversified financial services institutions in the world. Operating globally through the J.P. Morgan and Chase brands, the firm provides a comprehensive range of financial solutions including investment banking, consumer and commercial banking, transaction processing, and asset management. Serving millions of consumers and many of the world's most prominent corporate, institutional, and government clients, the company combines deep market expertise with extensive technological investments.
- Industry
- Banking
- Company size
- 320000+ employees
- Founded
- 1799
- Location
- New York, New York, USA
- Funding stage
- Public Company
Leadership
Chairman and Chief Executive Officer
Chief Financial Officer
Chief Operating Officer
Global Chief Information Officer
CEO, Asset & Wealth Management
Recent coverage
GuruFocus
JPMorgan Chase CEO Warns of AI Cybersecurity Risks as JPM Retains Top AI Bank Ranking2026-10-06
Zacks Investment Research
JPMorgan Chase & Co. (JPM) Reports Next Week: Wall Street Expects Earnings Growth2026-10-06
JPMorgan Chase Newsroom
Ford Motor Company, JPMorganChase, State of Michigan, Michigan Central and Newlab launch Michigan LIFT2026-09-29
JPMorgan Chase Newsroom
JPMorganChase expands DFW presence with new Collin County office2026-08-27
JPMorgan Chase Newsroom
JPMorganChase doubles down on housing, aiming to deploy $750 billion through 20352026-08-03