XPENG is a leading smart technology company at the forefront of innovation, integrating advanced AI and autonomous driving technologies into its vehicles, including electric vehicles (EVs), electric vertical take-off and landing (eVTOL) aircraft, and robotics. With a strong focus on intelligent mobility, XPENG is dedicated to reshaping the future of transportation through cutting-edge R&D in AI, machine learning, and smart connectivity.
The Mission: We are building the next generation of end-to-end autonomous driving on physical AI foundation models — vision-language-action policies and world models — and the same training stack carries our robotics program. These training runs span thousands of accelerators and days of wall-clock — how fast we train them is how fast we learn. As a Senior Staff Machine Learning Performance Engineer, you will own that speed: working with the modeling teams on their hardest bottlenecks, setting the technical direction for performance, and turning one-off wins into defaults — including the agentic workflows that find the next round of them automatically.
Job Responsibilities:
Training Performance at Scale: Own step time, scaling efficiency and long-run stability for foundation-model training across thousands of accelerators — from the data path to kernels, parallelism and communication.
Cross-Team Technical Leadership: Work directly with the modeling, infrastructure, simulation, data-mining and robotics teams on their hardest bottlenecks, and set the performance direction those teams build against.
Reusable by Design: Turn one-off wins into modular components that become defaults across most models and workloads, rather than a patch that helps a single job.
Observability at Scale: Extend performance tooling and metric coverage across the stack, while keeping instrumentation overhead low enough to stay on for production jobs fleet-wide.
Agentic Automation: Build agents that reproduce any job on demand, verify numerical parity and separate a real regression from run-to-run noise — so that most of the optimizations that land are found and fixed by AI rather than by hand.
Minimum Skill Requirements:
Master's or PhD in CS, CE or EE, or equivalent practical experience, with 8+ years of industry experience, the majority of it in performance engineering.
Hands-on experience optimizing GPU training or inference workloads in production, at multi-node scale.
A track record of diagnosing performance problems in real production workloads and landing the fixes — often in systems owned by other teams — with the gain measured and verified in production.
In-depth understanding of computer architecture and high-performance computing.
Strong C++ and Python, with a solid command of data structures and algorithms.
Preferred Skill Requirements:
Hands-on experience writing and tuning CUDA or Triton kernels.
Fluent in PyTorch, with concrete insight into what makes PyTorch code fast or slow — graph breaks, host-side overhead, synchronization points and memory layout.
Training at the scale of thousands of accelerators, including multi-node failure modes and long-running job stability.
Building profiling, tracing or benchmarking infrastructure that other teams depend on — CUPTI, Perfetto, or equivalent.
Using LLM agents to automate parts of an engineering workflow.
Autonomous driving, robotics, or another domain with hard latency, power and memory constraints.
Publications or open-source contributions in ML systems or performance.
What do we provide:
A fun, supportive and engaging environment.
Infrastructures and computational resources to support your work.
Opportunity to work on cutting edge technologies with the top talents in the field.
Opportunity to make a significant impact on the transportation revolution by the means of advancing autonomous driving.
Competitive compensation package.
Snacks, lunches, dinners, and fun activities.
The base salary range for this full-time position is $244,140, - $413,160, in addition…