Codex Engineering

Thank you all for being here. The room is a lot larger than it was the last time I was here, six years ago, so I'm very grateful for that.

As mentioned, I work on Codex, which is OpenAI's coding agent. One thing that stands out for context: looking at last year's ML for Systems Conference, my colleague Ahmed was here talking about how OpenAI models were starting to get capable enough to really get gold at the IOI or Olympiad-level competitions. But I think it's only really this year in 2025 that we started to reach the point where these models could actually start to do real-world software engineering work in a way that was very meaningful for users — which many of you have probably already experienced.

This chart shows a benchmark called SWE-Bench Verified. This is from fine people at Princeton, many of whom are at the conference working on much better and newer benchmarks, but SWE-Bench is still a great example. If you're not familiar: it's a benchmark of 500 tasks from open-source issues in GitHub repos, and the models need to produce the pull request that resolves the issue.

Even last year, around June, when GPT-4o came out, the models were still not very good at this task — starting around, I think, between 30 and 40%. But just in the last year, the progress has been pretty incredible. A lot of it's been driven by developments in test-time compute, as we heard in the previous presentation. With these reasoning models, the progress has really taken off. The models can resolve almost twice as many issues as before; we're really nearing the 80 to 90% mark, which is probably pretty close to the limits as far as this benchmark goes.

Looking at this METR time-horizon graph — shared around a lot recently — it's not just about completion, but also about the length and difficulty of the tasks models can achieve. It's almost like an exponential curve: the rate of progress in how difficult the tasks models can solve is increasing over time. I think the Codex models are actually pushing the frontier of that right now.

Going back to the IOI thing: this year the models achieved some very impressive feats — not just gold at the IOI, but also the IMO, and a perfect score at the ACM ICPC finals. This is actually the same underlying reasoning model that powers Codex. So when you use Codex — and I hope you try it if you haven't already — you're getting this level of intelligence and reasoning at your fingertips, whether that's on the computer or in the cloud.

Codex was released — the CLI was released earlier this year, in April. You can install it right away; it's a simple npm install. It's also available in the IDE, and as the cloud product in ChatGPT, which was released in May. What's cool is that the same underlying open-source agent — we've released it on GitHub as openai/codex — is open source. All that core logic is the same underlying agent infrastructure that powers the different product surfaces. We're releasing that to the public so anyone can achieve kind of the same level of performance that we do.

The models are also available in the API. I think the 5.1 Codex max model just became available in the API yesterday. And we're really starting to see adoption take off both internally and externally. There's no better evaluation than real-world usage. Something like 95% of technical staff at OpenAI make use of Codex. Codex reviews 100% of pull requests created at OpenAI. Externally, usage as measured in daily messages to Codex has increased about 20× since GPT-5 was released in August — which was kind of the first OpenAI model specifically trained to be good at these coding-agent scenarios.

What's under the hood of the Codex agent? We've intentionally kept the core agent infrastructure — the loop that powers the agent — extremely simple. The nice part is it's open source. There are links at the bottom; you can go to the repo and check these out. We've open-sourced the prompts, the tool specs, all of that.

When you start a task, in the prompt you basically have the developer instructions, a set of function-calling tools we provide to the agent, the environment / user's environment context, and of course the prompt the user gave us. Then we have what is ultimately quite simple: a loop where the model receives that prompt, reasons about what to do next, and almost always makes a tool call — running a command, or using an MCP if you've provided a Model Context Protocol tool. Afterwards we provide the agent the results of that tool call. This whole process repeats until completion.

Essentially a single for loop. We use the Responses API, or you can also run the GPT-OSS models locally. You just repeat this in a loop until the task is complete. Ultimately that's all you need to achieve both these incredible performances at competitions and solving real-world software engineering tasks.

Underneath that simple loop, there's a lot going on. I want to provide a high-level overview of some of the challenges we've solved — interesting because it's at the intersection of ML and systems. First: high-quality environments, not just for training the models but also to serve them as part of the Codex cloud product. Second: a lot of thought goes into the tools the agent gets. Third: constraints around model behaviors, durability, and what it means to have a coding agent users actually want to use — not just things that are incredibly capable but otherwise hard to work with.

Environments. If you go to the Codex cloud product, you can create your own environment. The cool thing is we've taken the same infrastructure that powers our training environments and shipped it as the Codex cloud product. A lot of the knobs you see when you create a Codex environment as a user are the exact same types of things we use during training. You start with a container image, usually a Dockerized container; it might contain a mix of pre-installed dependencies, but users can also provide their own.

Security matters. By default, most containers have internet access disabled, but users can selectively enable internet. Many enterprises have concerns around data filtration, or accidentally installing bad software or bad dependencies from the internet. It's important to have those knobs, and also dedicated training so the model is less vulnerable to that type of injection.

Tools and agent loop. We have a fairly opinionated take on what tools you should give the agent. Our philosophy is that the terminal is the bread and butter of what Codex uses to accomplish most tasks. We do think that for a lot of things, the terminal is kind of all you need. That's not true for all tasks, of course, but we find it true for the majority. You can read files, write files, run tests, review and commit changes — the fundamental inner loop of the software development cycle.

Below is roughly the interface we use for the exec / terminal tool. A couple of callouts: we expose knobs that let the model control how much token usage any tool call takes and how long that tool call is allowed to take. The model has fairly fine-grained control over this cost and latency tradeoff, and through reinforcement learning the model can almost learn by itself how to best optimize that.

Another philosophy: enable sandboxing by default. On macOS and Linux there are sandboxing primitives where you can execute processes in a pseudo-sandbox where file and network access is tightly restricted by default. My colleague Fouad has an excellent talk about considerations that went into this design. On macOS you can define sandboxing policies, similarly on Linux; we have a fairly well-thought-out set so the agent can't just delete your home directory without explicit permission — unless you wanted that for some reason.

With the terminal tool alone, there's a benchmark called TerminalBench where the agent has to do various things — not just pure coding tasks, but start a server, download videos from the internet, and so on. With that simple agent loop and a relatively simple interface to execute commands, we're already able to achieve state-of-the-art performances on these real-world tasks that require interaction with a computer.

One interesting anecdote: you would think models can just use Git to apply patches or edit code. It's actually a pretty interesting design space — how do you provide models tools to edit files? Git diffs have a big advantage: they're well represented in the pretraining data. One downside is that Git diffs contain a lot of references to line numbers, which make it very difficult — basically impossible — to predict up front in an autoregressive manner. So we have a custom patch format; we've shared that with our developers. It's basically a simplified version of the Git diff format, no line numbers, but still the same approximate format: headings, context, and the patch itself at the core. During serving, the grammar of the patch syntax is extremely simple, so you can use constrained sampling to ensure the model always generates a syntactically valid patch.

Behavior and durability. This is a big and important piece of what makes coding agents useful in the real world. One thing we've worked with other companies to try and standardize is the concept of AGENTS.md — basically a readme for agents, the same way we have contributor guides for humans. The idea is we provide the model with guidelines around code structure, build and test commands, style guidelines, how to make a PR in the repo — and we actually enforce that the model adheres tightly to these instructions.

One interesting thing: the models are actually very good at creating these files. For example, a model like OpenAI research-class models is quite good at that out of the box. One trick: for many open-source repositories, it's possible to get a pretty good AGENTS.md out of the box just by deploying our own agents to do that.

I touched on the benchmarks earlier, but one limitation of a lot of the current set is that they only capture correctness as measured by the ability to pass the reference unit tests. In the real world there's so much more to software engineering than correctness: coding style, engineering best practices, when not to repeat yourself versus when it's okay to duplicate code — a very nuanced decision — how to structure code to be modular, even usage of comments, overly defensive programming patterns. When it doesn't go right — this picture on the right is actually from a Twitter comment — sometimes models are prone to doing things that look a bit AI. If we don't do our jobs right, that kind of leaks out. It's best to try and mitigate these types of things.

Another interesting thing: Codex is actually a great code reviewer by itself. On GitHub, if you are connected to Codex, you can just @Codex and ask it to review your PRs. As I mentioned earlier, all PRs at OpenAI are reviewed this way. There's been a surprising number of actually quite serious issues that have been flagged early as a result. We've published a blog post recently on the new OpenAI alignment blog, which I recommend checking out. There's been a lot of work that's gone into making the models very good at code review.

One concern specifically for code review — if you've used other code review tools — is a very delicate tradeoff between precision and recall, or the signal-to-noise ratio. If you flag things that are not true issues, that often wastes more time than you actually save the user. You have this very delicate tradeoff between precision of the flagged issues versus being able to flag the actual issues that occur in practice. We've worked a lot to really tune that — not just find the right spot on the curve, but also push the curve upward, as you can see in this plot.

There's a lot of safety training that goes into our models — not just Codex, but all OpenAI models. Some things specific to coding agents: prompt injections, not just through user input, but also files in the filesystem or even dependencies, where developers might try to sneak in attacks against LLM agents. One area adversaries in the wild are actually using or trying to use these models today is generating malware or trying to exploit cybersecurity risks — we actively work to mitigate that. Also jailbreaks and general model safety training.

One interesting callout from our system card: cool techniques we can use to make the models more collaborative. Previously we saw models prone to stepping over user changes if you're editing a file at the same time as the coding agent. You end up with conflicts that step over each other's work. We actually simulated during training users — another model would be inserted in the loop and make changes at the same time as the original model. By doing so, you can simulate cooperating with the user on your laptop and make sure the model behaves responsibly when there are conflicting changes.

What's next. One thing from the previous presentation that's very relevant for coding agents: using parallel test-time compute to improve performance. We published this number with the Codex-1 model — a bit old, but the trend still holds — the more attempts you give these models, you get a pretty smooth scaling curve in accuracy. The model might not solve the problem the first try, but if you give it four tries or eight tries, the probability of a correct solution increases quite a lot. We've actually exposed that in the Codex cloud product: when you provide a task, you can ask for multiple responses.

Another interesting technique is to extend the context of these models via compaction. The model can summarize its work, often compressing the tokens used in the context by up to a hundred times.

To close: the best part about working on Codex is that as we get a better Codex, it makes the future of Codex models better — faster to develop, whether through better tooling for the research side or faster development on the Codex product. Codex can start to review its own code; Codex agents can even help us make more tasks in training environments; and ultimately our purpose as a company toward our goal of accelerating progress toward AGI.

That's all I have. Thank you for listening.

Q&A

Mertz (BrowserUse): There are many components to training — safety, aligning with real-world usage, RL on typical tasks. How do you reconcile all of these at scale? You can generate a lot of test cases you can run and verify, but the same is not true for safety or alignment for collaborating with humans. Is it all combined into one pipeline, or stages separately?

Speaker: We're all about things that scale well. As much as possible we try to address all of these things at the same time. Ultimately our north star is aligning these things with how the tools are used in the real world — capturing how people actually use the product in practice and making sure the training reflects that real usage.

Eileen (ETH Zurich): Do you think there's a problem with this compaction method? Do we need new solutions? I assume it's some kind of summary — what do we put into the summary? Do we miss relevant information? Are you rethinking strategies?

Speaker: There's definitely room for more sophisticated strategies, but we find this approach works well. One of the powerful things is that it's basically the model that decides what to put in the summary. Over the course of training, the model gradually learns how to do this on its own, which is pretty interesting to see.

Shabas Patel (Best Buy): When you're talking about parallel test-time compute scaling, how do you plan to do the verifiers so they can also scale at the same time? As a user, how can you scale the verification process?

Speaker: I think that's an area that we're actively looking into.

Pranav (startup): You mentioned the token consumption is dynamic — the model learns how to consume more tokens or less. Is that applied at a global level or personalized for every user based on their coding patterns? How many tokens does it consume — personalized per user? RL on a per-user level?

Speaker: The one I showed specifically is at the tool-usage level. When a model runs a command, it can control the maximum number of tokens it should see from that tool call. It depends on the command: if you're running a test command, you might see a lot of output, and it might make sense for the model to be a little more restrictive about how many tokens should be seen from the result.

Cool. Thanks.