Intro

I recently had the pleasure of visiting OpenAI’s offices in San Francisco. Had a great time there, and it was a pleasure getting to know and talk to different engineers and engineering leaders.

Here is also the article, based on my conversation with Michael Bolin, Tech Lead for the Codex open-source repo. We talked all about how he does AI-assisted engineering:

How OpenAI Codex Tech Lead Does AI-Assisted Engineering Gregor Ojstersek · Jun 4 Read full story

In today’s article, I am sharing the insights from my conversation with Venkat Venkataramani, VP of Applied Infrastructure, OpenAI, and Emma Tang, who is leading Data Infrastructure at OpenAI.

We talked all about how infrastructure teams work.

This is an article for paid subscribers, and here is the full index:
- 5 Core Pillars of Infrastructure at OpenAI
- Infrastructure teams are critical to the success at OpenAI
- The amount of AI-generated code is lower than with product teams
- How they prioritize work?
- Unblocking teams is a crucial part of prioritization
🔒 Their primary challenge with infrastructure at OpenAI has always been the pace of growth
🔒 Another challenge is the unpredictability
🔒 AI-generated PRs have also shown to be another infrastructure challenge
🔒 What makes a great engineer for an infrastructure team?
🔒 How they measure the success of the infrastructure teams?
🔒 Last words

Let’s start!

5 Core Pillars of Infrastructure at OpenAI

At OpenAI, they organize their infrastructure around 5 main pillars, each focusing on a specific part.

1. Developer Productivity

Focusing on improving the software development process + research, giving everyone the tools and capabilities to do the best work possible.

Think of it as everything that helps teams build, test, deploy, and move faster, so that everyone can move as fast as they can.

2. Compute Infrastructure

This is the specific hardware and/or software needed for all the computing behind OpenAI’s products. This includes the infrastructure powering: ChatGPT, API, Codex, and any future products.

Venkat also mentions that this specific pillar could be referred to as Fleet Infra, as it’s specifically focusing on GPU infrastructure.

3. Networking and Storage

With this infrastructure, AI workloads can communicate while storing data generated by models and apps.

This pillar goes hand-in-hand with the compute infrastructure, as their work is closely related.

4. Observability

Monitoring, telemetry, metrics needed to keep the systems reliable. That’s the focus of this specific pillar. The goal is to detect issues quickly, understand what’s happening across the infrastructure, and ensure that the services are reliable.

5. Data Infrastructure

The last but definitely not least. Venkat describes it as one of the most important infrastructure organizations because basically every application continuously generates data.

This is the infrastructure that enables OpenAI’s feedback loop, helping improve future models and products.

Emma shared a great example of their work, which is an in-house data agent. They created an internal-only tool where everyone inside OpenAI can ask data-related questions and get the insights needed in order to make data-driven decisions.

No need to go to data engineers to “generate another dashboard”, they can simply ask the data agent to get the desired insights they are looking for.

Infrastructure teams are critical to the success at OpenAI

Everyone’s productivity is determined by how well the infrastructure enables them.

With bad infrastructure, you are limiting how fast people can move and make things happen. You want to enable everyone instead to do the best work possible.

It’s crucial to enable both:

• teams and people internally at OpenAI, and

• the customers/users

to be able to do what they want to do.

A good example is the following: Good infra plays a huge role in user experience, like how fast ChatGPT responds to the questions being asked, and also how reliable it is.

And there’s another important factor as well, which is cost, and balancing cost with speed and reliability. How to find the right balance between them, so that they don’t burn the full budget just on compute.

And then also specifically for data infrastructure, it’s important to have a good infrastructure in place so that every team inside OpenAI is able to make data-driven decisions.

It’s important to view infrastructure as the backbone of many other essential functions a business can’t operate without.

The amount of AI-generated code is lower than with product teams

This is a very important insight, mentioned by Emma, as many people may believe that you can just use AI and generate thousands of lines of code without consequences. That’s completely not the case when you are building infrastructure.

The infrastructure work is a lot more critical to the business, and with one simple mistake, there can be huge consequences. You can block other teams from being able to progress with their work, and even worse, the users might not be able to achieve what they want to do using their products.

They focus a lot more on understanding every single line of code and making sure that everything works correctly. Therefore, they don’t see as much higher productivity boost using LLMs than some other more product-facing teams.

How they prioritize work?

As you may know, working in any kind of platform/infrastrucure team, there’s always a big issue with how to effectively prioritize the work, as there are often many competing priorities. I’ve asked Venkat about how they think about prioritization.

The really important message and how they think about infrastructure is the following.

They think about infrastructure as a product.

So, the main thing they keep in mind is that any kind of infrastructure you are building, e.g., CI/CD pipelines, data infrastructure, analytics, cloud infra for compute, etc., think of that as a product that you are building.

And everyone using your infrastructure (product) is the customer. Very similar to having an actual product and offering it to real customers.

And an example that Venkat mentioned:

If you are a startup, and you have a lot of competing customers, and you want to say “yes” to all of them, you find a way to make it work.

That’s very similar to what they try to do at the infrastructure organization as well. They try to say “yes” as much as they can, but oftentimes there has to be a certain negotiation.

They might not be able to do everything that a certain team asks for, but maybe they can do 4 out of 5 things, and that can be delivered in the following week.

Unblocking teams is a crucial part of prioritization

And the second important thing regarding prioritization is that if a certain ask is far away from their focus, they’ll ask how they can unblock that specific team as quickly as possible, so that at least they can move forward.

And they’ll focus on building a quick solution (even if that might not be the best possible solution), just to unblock that specific team. What’s important here is that the solution is built in a way that can be easily replaced with the proper solution down the line.

And what they’ll do after is launch a project where they would find the best solution to that specific ask, and build it in the next 1-3 months.

Think of it as building an MVP → unblocking the team with it → building an actual long-term solution and replacing the MVP.

Their primary challenge with infrastructure at OpenAI has always been the pace of growth

Venkat mentioned that building infrastructure that scales to these heights is a hard but not impossible task. ...