Research Engineer, Collective Alignment

About the team

OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity. OpenAI’s Collective Alignment Team works on technical approaches to ensure that AGI is shaped democratically. We plan to do so by implementing a system for collecting and encoding public input on model behavior into our systems. While OpenAI has existing efforts to solve technical alignment, we also need systems to help us determine the answer to the question “What do we align to?”

Regulatory bodies are building up capacity to manage such systems, but may not be up-to-speed on the latest technology. Our best bet is to find more ways to ensure that we are appropriately managing risk and harms that our technologies may pose, including by increasing our ability to make good decisions about system development and deployment. 

One way to increase our ability to make good decisions is to have input from the broader public. We are encouraged by the phenomenal work done by the recipients of our Democratic Inputs to AI grant that we launched in May, which was aimed at exploring and building in the space of governing AI behavior democratically. We selected 10 teams (out of nearly 1000 applicants) who were each awarded $100,000 to build, test, and share learnings from projects that explore how we can use democratic methods to decide the rules that govern AI systems. 

We’re excited to combine our research with ideas and prototypes developed by these teams in the coming months. We will implement a system that can find representative participants to provide their perspectives on a given topic, enable discussion and deliberation, and aggregate inputs into policies that are used to change our model behavior. And we’re looking for excellent, highly motivated Research Engineers to help us.

About the role

We are seeking Research Engineers to help design and implement experiments for collective alignment research. Responsibilities may include:

• Writing performant and clean code for ML training.
• Independently running and analyzing ML experiments to diagnose problems and understand which changes are real improvements.
• Writing clean non-ML code, for example when building interfaces to let workers and participants interact with our models or pipelines for managing human data.
• Collaborating closely with a small team to balance the need for flexibility and iteration speed in research with the need for stability and reliability in a complex long-lived project.
• Understanding our high-level research roadmap to help plan and prioritize future experiments.
• Implement experiments to measure the effectiveness of different preference learning techniques, including RLHF 
• Examine the impact of different aggregation methodologies on model behavior
• Curate large datasets of prompts and investigate coverage of boundary cases
• Exploring methods to understand and predict model behaviors, such as finding inputs causing anomalous circuits or catastrophic outputs
• Designing novel approaches for using LLMs in democratic inputs to AI  research

You might thrive in this role if you

• Are excited about OpenAI’s mission of building safe, universally beneficial AGI and are aligned with OpenAI’s charter
• Want to use your engineering skills to push the frontiers of what state-of-the-art language models can accomplish
• Possess a strong curiosity about the sociotechnical challenges around aligning and understanding ML models, and are motivated to use your career to address this challenge
• Enjoy fast-paced, collaborative, and cutting-edge research environments
• Have experience implementing ML algorithms (e.g., PyTorch)
• Can develop data visualization or data collection interfaces (e.g., JavaScript, Python)
• Want to ensure that humanity can shape future powerful AI systems