Principal Data Engineer (PySpark, AI)
Autodesk · Posted Oct 28
3D design, engineering, architecture, and entertainment software solutions
Get a personal compatibility score
Add a resume for personal matches
About the role
The Advanced Compliance Products (ACP) team is a strategic initiative focused on detecting non-compliant usage of Autodesk software through large-scale data processing and analytics. The role sits within a data engineering platform domain built on modern cloud technologies and AI-driven engineering practices. As a Principal Data Engineer, you will drive the technical architecture, scalability, and modernization of ACP's data engineering platform while remaining hands-on.
What you will do
- Define and drive the architecture of scalable, reliable, and cost-efficient data platforms processing billions of records.
- Lead the design, development, and optimization of large-scale data pipelines using Spark, PySpark, Python, and SQL.
- Drive platform modernization initiatives, including incremental processing, reusable frameworks, and modern data lakehouse architectures.
- Leverage AI/GenAI technologies and AI-assisted development tools to improve engineering productivity, automate workflows, enhance data quality, and optimize pipeline operations.
- Identify and resolve complex performance, scalability, reliability, and data quality challenges.
- Establish engineering best practices for CI/CD, automated testing, observability, and cloud infrastructure.
- Lead technical design discussions, influence architectural decisions, and mentor engineers across teams.
- Collaborate with product managers, data scientists, and business stakeholders to translate business requirements into scalable data solutions.
- Take end-to-end technical ownership of complex, cross-team engineering initiatives.
Skills used in this role
What the employer is looking for
- 10+ years of experience in data engineering, distributed systems, or large-scale data processing.
- Strong hands-on expertise in Apache Spark, PySpark, Python, Spark SQL, and SQL, including performance tuning and optimization.
- Proven experience architecting and scaling complex data pipelines and distributed data processing systems.
- Strong knowledge of AWS, Hive, Snowflake, and modern data lakehouse architectures.
- Experience with data modeling, ETL/ELT frameworks, orchestration tools (such as Airflow), and CI/CD practices.
- Hands-on experience with AI/GenAI technologies, including LLMs, prompt engineering, and AI-assisted development tools such as GitHub Copilot, Cursor, or equivalent.
- Practical experience applying AI to data engineering workflows, such as code generation, automated testing, data quality validation, anomaly detection, pipeline optimization, or intelligent troubleshooting.
- Strong understanding of data quality, observability, reliability, and cloud infrastructure.
- Demonstrated technical leadership, architectural decision-making, and mentoring experience.
- Excellent problem-solving, communication, and cross-functional collaboration skills.
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
Preferred qualifications
- Experience with Apache Iceberg, incremental data processing, and modern orchestration frameworks.
- Experience modernizing legacy data platforms and optimizing large-scale Spark workloads.
- Familiarity with BI tools such as Power BI or Looker.
- Experience with AI agents or LLM-powered automation.
- Experience building reusable data engineering frameworks and driving engineering standards across teams.
Benefits and support
- Annual cash bonuses
- Stock grants
- Comprehensive benefits package
About Autodesk
Autodesk is a global leader in 3D design, engineering, and entertainment software, providing solutions for architecture, engineering, construction, product design, and media creation. Its flagship products, such as AutoCAD, Revit, and Fusion 360, empower professionals across industries to imagine, design, and make a better world. As a pioneer in the space, the company continues to drive innovation through cloud collaboration, digital twins, and AI-powered design workflows.
- Industry
- Software
- Company size
- 10001+ employees
- Founded
- 1982
- Location
- San Francisco, California, USA
- Funding stage
- Public Company
Leadership
President and Chief Executive Officer
Executive Vice President and Chief Financial Officer
Executive Vice President and Chief Technology Officer
Executive Vice President and Chief Operating Officer
Chief Marketing Officer
Recent coverage
Autodesk News
Autodesk names Diana Colella to lead architecture, engineering and construction business2026-10-06
PR Newswire
Autodesk Advances Agentic AI in Its Three Industry Clouds2026-09-15
PR Newswire
Autodesk Enters Definitive Agreement to Acquire MaintainX for $3.6 Billion2026-05-28