We are seeking highly skilled engineers with expertise in machine learning, distributed systems, and high-performance computing to join our Research team. In this role, you will collaborate closely with researchers to build and optimize platforms that train next-generation foundation models on massive GPU clusters. Your work will play a critical role in advancing the efficiency and scalability of cutting-edge generative AI technologies.
Key Responsibilities
- Scale and optimize systems for training large-scale models across multi-thousand GPU clusters.
- Profile and enhance the performance of training codebases to achieve best-in-class hardware efficiency.
- Develop systems to distribute workloads efficiently across massive GPU clusters.
- Design and implement robust solutions to enable model training in the presence of hardware failures.
- Build tools to diagnose issues, visualize processes, and evaluate datasets at scale.
- Optimize and deploy inference workloads for throughput and latency across the entire stack, including data processing, model inference, and parallel processing.
- Implement and improve high-performance CUDA, Triton, and PyTorch code to address efficiency bottlenecks in memory, speed, and utilization.
- Collaborate with researchers to ensure systems are designed with optimal efficiency from the ground up.
- Prototype cutting-edge applications using multimodal generative AI.
Qualifications
- Experience:
- 3+ years of professional experience in ML pipelines, distributed systems, or high-performance computing.
- Hands-on experience training large models using Python and PyTorch, with familiarity in the full pipeline: data processing, loading, training, and inference.
- Proven expertise in optimizing and deploying inference workloads, with experience in profiling GPU/CPU code (e.g., Nvidia Nsight).
- Deep understanding of distributed systems and frameworks, such as DDP, FSDP, and tensor parallelism.
- Strong experience writing high-performance parallel C++ and custom PyTorch kernels, with knowledge of CUDA and Triton optimization techniques.
- Bonus: Experience with generative models (e.g., Transformers, Diffusion Models, GANs) and prototype development (e.g., Gradio, Docker).
- Technical Skills:
- Proficiency in Python, with significant experience using PyTorch.
- Advanced skills in CUDA/Triton programming, including custom kernel development and tensor core optimization.
- Strong generalist software engineering skills and familiarity with distributed and parallel computing systems.
Note: This position is not intended for recent graduates.
Compensation
The salary range for this role in California is $175,000–$250,000 per year. Actual compensation will depend on job-related knowledge, skills, experience, and candidate location. We also offer competitive equity packages in the form of stock options and a comprehensive benefits plan.