Agenti umetne inteligence, ki se sami izboljšujejo: Exo in grenka lekcija | Alex Krentsel

Vir, odlomek za odlomkom

1. u1 0:00 What makes a harness "bitter lesson aligned"?
2. u2 0:00 As models keep getting better and better, the big question is: What is going to stand the test of time? That is, what kind of a harness design is bitter lesson aligned? And the answer is, we have to look at harnesses that have a new set of design goals, that are flexible, that are legible, that are editable. Where as the models get better, the harness can get better as well.

   

3. u3 0:21 And to do that, the harness has to be able to edit itself to specialize to the task at hand. But to do so safely, in a way where we can trust that it's not going to break itself or lose its state.
4. u4 0:31 Introducing Exo and how the project started
5. u5 0:31 Yeah, so I'm excited to present today on, our system called Exo. And so the focus of my talk is going to be how we should think about architecting systems that are able to improve themselves at runtime. This is joint work with an interesting cast of characters. My advisors, Sylvia and Ion, who I work with very closely now.
6. u6 0:50 but you'll also notice Martín Casado and Ankur Goyal here. Martín, is like a deep systems person, even though he's a VC. and he came and gave a talk at Berkeley on where he was seeing agents going, and this was around the time when I was also giving a talk, and I posted a talk online on, OpenClaw, and the internals of how OpenClaw works. And that talk got some attention.
7. u7 1:11 He saw that. I went and shared the slides with him, and we got together and started brainstorming about what the future of a system like OpenClaw could be. So we started hacking on this together. So this has kind of been a joint work across industry and academia.
8. u8 1:24 Background: from next-token predictors to autonomous agents
9. u9 1:24 So just to set the scene a little bit, I think we're all very well aware, every day there's more and more model releases. Th-these are just from the last month alone, from, September. and each time they come out, there's a huge focus on how big or fast or performant each of the new models is. and we see these exponential curves that we're clearly on an exponential curve,

   

10. u10 1:44 especially now in twenty twenty-six. But the model is empowered by the system that it's put within. So the model is only as good as the system it's in. And so these models act on the world through agentic systems. Maybe the most common ones we're familiar with are Codex and Claude Code. There's many others.
11. u11 2:03 But the thread throughout my talk, the open systems question, the research question that drives me is: What are the right primitives and principles for building agentic systems that are going to maximally empower our models to be effective in the real world? That's gonna be the through line of the talk I have at first a few slides on background of kind of where we are.

    

12. u12 2:25 Now, I know you guys know this here at Arena, but these slides I originally made for kind of a broader audience. But just to set the scene so we're on the same page, this is how I see history a little bit, which is we had a phase zero where we really saw LLMs as next token predictors. And I was graduating in twenty nineteen when this happened. I remember Google's BERT, OpenAI, G-- OpenAI's GPT early models.
13. u13 2:47 Google had LaMDA. These are really next token predictors. and then in twenty twenty-two with the launch of ChatGPT, we saw that if we fine-tune these next token predictors, we could get them to act very effectively as assistants. We had ChatGPT, Claude, and Gemini. And then there was this middle period that was really interesting
14. u14 3:05 of LLMs given certain tools and orchestrated upfront by a human to act in a certain series of ways. Kind of almost a model of one action passed to another action passed to another action. And frameworks like LangChain or AutoGen, Cr-- CrewAI, what I would kind of call scoped agents.
15. u15 3:24 It was a very clunky time. and finally, in phase three, which is the tail end of where we're at now, and we're about, I think, to enter phase four, we moved to LLMs that had tools but that could dynamically decide which tools to use at runtime. And kind of giving them that unconstrained ability to make those

    

16. u16 3:43 decisions is what makes these agents more autonomous than the previous state that we were coming from. this is Claude Code and OpenClaw and other things. yeah, so we're here. So at the end of the day, all of these systems, when you look at them, boil down to LLM calls. and there's like this agentic system loop.
17. u17 4:02 And the difference between these systems is what context is provided to the actual LLM. And so Again, this will be one of my last kind of background-y slides, but you have your transformer inference, which predicts one token at a time. But then a large language model is repeatedly sampling, repeatedly asking the transformer for the next token over and over, taking the previous all the tokens
18. u18 4:24 you have and asking for the next one. And then we've had assistants which repeatedly sample, ask an LLM to give a response and interleave it with user messages. and then we got our scoped agents like Claude Code and Codex and Cursor that go and repeatedly use kind of an assistant but also provide tooling and are able to
19. u19 4:43 execute tooling and interleave kind of observations and actions on the world in with the repeated calls to an assistant. then I think we got to, earlier this year, a layer of an autonomous agent kind of, that owns its environment a little bit more than Claude Code. Claude Code, you drop into a directory, and you kind of
20. u20 5:02 are working in the directory. OpenClaw lives at a level of like it kind of owns almost a computer or a server, and it acts on multiple agents. It can spin up multiple agents, multiple threads in parallel. And the emerging new layer that I think is about to be really big, we're on the cusp of really big things happening here, is a self-improving layer.

    

21. u21 5:17 The emerging self-improving layer
22. u22 5:21 So it's yet one more outer loop that allows, an autonomous system to kind of go and edit its own policy, the tooling that it has, the environment that it owns, the policy by which it in- interacts with that environment. and to me, this is kind of like the most powerful loop, which is why it's something that I'm passionate about and something that I'm working on.
23. u23 5:43 yes, learning to improve. Okay. So through the rest of this talk, I'm going to talk a little bit about how agentic systems look today, at least in my vocabulary, the challenges of building these agentic systems. Then I'll present to you our approach called Exo, which is an architecture for self-improving agentic systems.
24. u24 6:03 and I'll close with some results on benchmarks and some really fun open world tasks I'm excited to show you guys. and again, please interrupt at any point, if you wanna discuss.
25. u25 6:14 What is an agentic system? Defining harness and policy
26. u26 6:14 So what are agentic systems? I think here we all know this is a really important emerging class, but what's really interesting is even when I go talk to experts at different companies, in academia, you talk to professors, the vocabulary everyone is using is still all muddled.

    

27. u27 6:30 It's all mixed up. What is an agent? What is a harness? what is an environment? these things mean different things to different people. So I'll present my definition, which is, to me, an agentic system is any system that has an AI reasoning component within it somewhere. It's kind of-- But really you can think about it somewhere where there's an LLM call. and, yeah, most common or frequent example of this is a coding harness.
28. u28 6:54 but, this kind of layer around the LLM call that supports it, the scaffolding, we call a harness, and I'm gonna my-- present my definition here. We can debate this, but to me, a harness has two and only two tasks. It is a black box that has a northbound interface that assembles context for LLM, an LLM call, and it has a southbound interface that acts
29. u29 7:18 on and observes some environment. Everything else is the black box. Whatever it does inside is up to the harness, and, these are the only two things that a harness kind of has to have abstractly. And I'm going to use the word policy to refer to all of the logic that has to do with what's inside of that black box. So anything to do with how you assemble your context, and act on an environment.
30. u30 7:41 now we're definitely most familiar with coding harnesses, but there's a wide range of domain-specific agents that are not just for coding. People are using agents to, run business queries, to handle insurance claims, for security, medicine, drug discovery, all these different things. and really when I look around at my friends, it seems like, you know, New
31. u31 7:58 York seems to be the like hub of companies that are kind of doing agentic vertical applications over top of some domain. SF seems to be the hub of like kind of cross-cutting core ML or, kind of agents development. That's the sense I get. but good policy decisions very clearly affect performance of our agents
32. u32 8:19 and how well they do at their tasks. Quick question.
33. u33 8:22 Q&A: domain-specific vs. general agents, thicker vs. thinner harnesses
34. u34 8:22 Yeah, please. So can you go to the be-last slide? Mm-hmm. So I know there's a lot of debate within like domain-specific agents or general agents. what's your take on? Because a lot of people think like if we keep improving the so-called reasoning capability of AI, what you just need to change is the environment tool. Mm-hmm.

    

35. u35 8:41 Like the tool to communicate between so there's no domain-specific agents needed. And, from a system side, what's your thought? And is it like anything that will change, given this different thoughts to impact the system design? And also like from a definition side, what's your, what's your, opinion on this? I actually have some slides on this on my prediction at the very end of my talk.
36. u36 9:04 So I will come back to this. But I, do wanna call out because it's an excellent question. There's a fundamental question people have about harnesses, which is as our models get better- Are our harnesses going to get thicker or thinner? Meaning, are we going to squeeze out more gains with time by adding more kind of custom functionality into the harness?
37. u37 9:21 Or do we need a really minimal thin harness that gets out of the way of a model, and the smarter a model gets, the more it'll eat into the harness's behavior? I have a thought on this, but let me save it till the end. Okay. But that is the question you're getting at, right? I think. Yeah. Yeah. Okay. now, good policy decisions affect the performance of our harness. and I just so that we make this very concrete, what is policy?
38. u38 9:45 To me, policy would be the tools that the harness, provides. Meaning, usually, if you look inside of a harness, it assembles in its prompt to, up to the LLM. At the start, it'll have some block of, like, system instructions, but then it'll have a list of tools, and it'll be like, "You can call this set of tools." that set of tools is part of the, policy of, the design of the harness. The way you assemble context.
39. u39 10:06 Do you take the last hundred messages, or do you take the last ten? More messages means more context, but it also costs more money. the prompting that you choose to put at the beginning, the me-- the way you index skills and memory, all these different things. Anything that's kind of inside of that black box that's static code, to me, is part of the policy. and I claim, a-and I think people would agree with this, the right
40. u40 10:29 decisions for what this policy looks like is pretty domain-specific. So if you poke your head inside of coding agents, you'll notice that they often have, some sort of LSP tooling, which is Language Server Protocol, which goes in as you're working on a code base, kind of builds an abstract syntax tree of your code, so that if you wanna rename variables or, move around or refactor code, it gives a kind of a graphical view to the agent,
41. u41 10:51 so it doesn't have to work over text. And so adding tools like that, definitely makes the agent more effective at being able to refactor your code. It's very different if you're then working on in bio or in system synthesis or for each of these domains, there's kind of particular tooling or particular policy decisions that are effective.
42. u42 11:09 So these are my premises.
43. u43 11:13 Why finding a good policy is hard: agent optimizers and their limits
44. u44 11:13 now, how we get to good policy is changing. before we even had harnesses, Actually, I'm curious if anyone remembers in '22. Did anyone get into, like, prompt engineering? You remember there was a whole craze for like six months. There were like websites where people would share their prompts, and then there

    

45. u45 11:30 were like guides on how to better prompt. I remember trying to generate like an image of a bear. and if you ask for the bear, it'll draw a cartoon bear. But if you say like bear, thirty-millimeter, high resolution- … soft fuzz, like hyper-realistic, then it actually looks really good. So there was a period of time where we were kind of acting as the machines that were producing prompts.
46. u46 11:49 and we were kind of learning how to specialize those prompts. And the way I think about a harness is kind of a system that's its job is to write a dynamic prompt and take action on an environment. But really it is the box that produces a prompt that goes to the LLM. but getting these harnesses right, picking the right tools
47. u47 12:07 for the domain isn't trivial. It requires domain knowledge. It requires time. And so there's been a lot of excitement in building these agent optimizers that try out different policies, that try to optimize an agent. Maybe the most known one was called, Meta-Harness from earlier in the year, from Stanford. GEPA does this kind of evolutionary work on prompts specifically,
48. u48 12:28 not agents al-altogether. AlphaEvolve, OpenEvolve, Sky Discover, EtaEvolve, EvoX is a bunch of papers that basically say, "Give me an evaluation data set of tasks you want your harness to be good at," and it will try different variants of a harness and reason about good ideas to add until kind of that metric goes up. So lots of excitement about having these outer optimization
49. u49 12:50 loops optimize our policy this sounds pretty good, but there are significant challenges with this. And I actually think here I won't have to convince you guys of them as much as I do when I talk to maybe other people. I'm-- I'll be curious if you agree or disagree with these. But for one, I think people assume that it's easy to get a dataset for
50. u50 13:11 the domain you're trying to work on. But actually, for many domains out there, you don't have a clean dataset that's collected, that's able to kind of evaluate how well you are at your task. So if you're an insurance company, just even getting and collecting a well-annotated dataset of the right behaviors in the right context to then try to optimize your system on is actually a huge feat.
51. u51 13:32 Once you have that dataset, it pretty quickly becomes outdated because your environment changes and the documents that you would collect change. and this is another thing that I'm seeing. We are increasingly asking agents to do longer and longer tasks. So in a world where each task was thirty seconds, you could go and try a hundred tasks on a harness variant and see how it did.
52. u52 13:51 But in a world where we're asking agents to spend days or even weeks to do tasks like proving a math theorem or synthesizing a system, building a system end to end, it becomes prohibitively expensive to try a hundred variants on a hundred tasks each to see how they do. So this outer loop optimization, I think is problematic. And the last thing in that diagram I had here, you pick some outer optimizer loop.
53. u53 14:16 You pick an agent builder, and there's all of these different papers and different techniques, and each of them claim to be better for particular domains. And so you're stuck with a fixed outer policy. You can optimize the harness, but the outer kind of way by which you're optimizing is fixed.
54. u54 14:30 Key idea: let the harness edit itself at runtime
55. u55 14:30 So observing these issues in this work, the question I'm asking is: Is there a simpler architecture that gives us the same degree of adaptiveness while alleviating these challenges? and the answer, the spoiler is yes, which we can do by turning to the reasoning we already have inside the agent.

    

56. u56 14:49 So the key idea we're going to have is a simple one, which is we want the harness to be able to edit itself at runtime. This is not a crazy idea. in fact, you might have predicted this idea based on what I've said before, but there's a cute reason why this is doable now at this layer, in the harness layer, where in many other places it's not.
57. u57 15:09 so I'll turn to control theory here. This is like a diagram of a adaptive control loop in control theory. And in this adaptive control loop, you have some actor, the harness policy, that takes an action u on an environment, and there are observations y that come out of that environment. And the harness gets to see those and keep taking actions until the observations match the reference that, it wants.
58. u58 15:29 Very simple model. And in these adaptive control loops in control theory, there's an optimizer that sits outside that gets to see the same action and environment result pairs and propose changes to the harness policy, so to the actor, theta. Now, what's really unique about the harness layer is two things. First, this property called homoiconicity, the word isn't
59. u59 15:53 important, but what is important is that the harness is code, and the harness also operates over code. So what that means is the tools that we often give to our coding harnesses, are, coding tools to read files, edit files, inspect code, reason over what code should be.
60. u60 16:10 and so the harness itself can reason about its own state. Why I think this is important, why I'm calling it out, is because this is not the case for models. If you think about an ML model, the weights for one of your models are way too big to fit into the context, like into the window that the model can take as input. So it's not feasible to ask the model to reason about its own weights.
61. u61 16:33 You can ask the model to propose ideas to how to train better, but you cannot actually take the model as input to the model. Whereas in the harness layer, you can take the code of the harness as input to the harness's reasoning because they're in the same language. So I think this is actually a really important thing. and the second is optimizer symmetry, which is that these actions theta,
62. u62 16:55 the way you would edit a harness, is in the same domain as the actions that your harness takes already. So a change to the harness is a delta in code. It's an edit to a file. And the harness already can edit files. So these two things make it so that we can collapse the optimizer policy inside of the actual core harness policy because that policy can make decisions that act
63. u63 17:20 on itself because the h- harness is code and the policy can edit code, and the optimizer actions are also code edits. Okay. This alone I think isn't a brilliant idea.
64. u64 17:37 The safety problem: fate sharing and unprotected state
65. u65 17:37 It's pretty clear you could ask a thing to edit itself. I think it's cool, but the real challenge here is a safety challenge. If you have a system that can edit itself, you bring in two new challenges. The first is fate sharing, which is if your optimizer breaks your system, accidentally makes a bad edit, when these were two separate things, in
66. u66 17:58 the next iteration, your optimizer could fix the system it just broke. But if you pull the optimizer inside of the agent, if the agent breaks itself, there is nothing left to fix itself. So you have a fundamental issue here. You have fate sharing, where if you crash your system, you have also crashed your optimizer. And the other is, your state, your history, conversation
67. u67 18:19 history, any snapshots you took, is also, uh, unprotected. So if you look at something like Codex or Claude Code today, you have a .claude directory or a .codex directory that has conversation history. And if your edits can delete your history, you can get stuck in situations where you corrupt-- Your policy may not go down, your process may not die, but you
68. u68 18:40 corrupt your own agent's behavior because the state has been erased, and you can't learn from state that's been erased. So just trivially rolling back your entire environment, you have to keep around the learnings of what you did wrong to not go down that path again. So you have to be able to reason more carefully about your, state. You have to separate the state from the actual policy as it's running.
69. u69 19:02 Exo's design: protected mechanism layer + evolvable policy layer
70. u70 19:02 and so how do we architect our system to allow self-improvement while avoiding these challenges? And this is where, Exo's approach comes in. So our design has kind of two key steps. This is what agentic systems look like today. I took all of these policy levers, and I put them inside

    

71. u71 19:20 of this ambiguous blob here. Our proposal is to cleanly partition an agent into actually two layers that have distinct and very important properties. There is a upper layer, which is the policy layer, which is mutable, and static, fully evolvable, and then there is a mechanism layer that is
72. u72 19:42 protected, isolated from any changes. And then our goal is to put the minimal functionality in the protected layer that enables or that gives us our safety guarantees while still exposing as much of the policy behavior up in the actual policy layer for self-editing. and our challenge is then to define which of this functionality should go into a
73. u73 20:06 protected layer to provide safety and the policy layer to be able to be evolvable.
74. u74 20:13 Inspiration from operating systems: the exokernel
75. u75 20:13 so here I'm gonna take one slide to give some context. I am actually not an AI person. I'm an, um, I'm a systems person. I spent my first three years doing networking. My advisor is Scott Shenker and Sylvia Ratnasamy. And so I'm, I'm a core networking person. and this debate isn't a new debate for people that come from systems. So there are decades of literature on operating systems.

    

76. u76 20:33 Operating systems deal with the exact same problem. We have applications and user space that wanna do things, and wanna be able to change the code. You wanna be able to run a-arbitrary applications, but then there's sets of resources that we wanna keep protected and make sure they don't get corrupted. And the way that operating systems deal with this is they have a notion of a kernel that provides particular sets of functionality and protects that
77. u77 20:56 functionality, while exposing a more safe amount of, kind of, change up in the user space for the application to write code. So when you write your code, you write Python or something. Python, you can express whatever you want in user space, but if you want to edit a file- You go through a file call, a, a kernel, a sys call into the kernel, which gives you a handle and an abstraction over a file.
78. u78 21:19 You don't actually get to edit memory. And the debate that the previous operating systems community has had is what the right layer is for placing this interface. What should be protected and what shouldn't be? And there was a very famous paper in the nineties called Exokernel, which made the case that we should put the absolute minimal, requirements,
79. u79 21:40 so just what's needed for secure-- keeping resources secure, down in the kernel and leave everything else up in the user space for use. so, and provide libraries for this. And so in this way, the application can choose a different file system or different virtual memory or a different networking stack
80. u80 21:59 while still protecting resources. and we're gonna follow the exact same, proposal here. and for the same reason. Exokernel makes this argument because it says this gives you policy flexibility to define whatever file system you want that might be more efficient for your application. I could give more examples, if people wanna talk about systems more.
81. u81 22:20 I love systems.
82. u82 22:21 What lives in each layer
83. u83 22:21 Okay, so here's our concrete split. inspired by this Exokernel design, in the protected mechanism, in that middle layer that cannot be changed or evolved, we put the state, which is conversation history, and this is really important, annotated with costs. So any resource usage gets annotated in this append-only log of what's happened.

    

84. u84 22:43 Okay? environment snapshots. So if you ever wanna snapshot your environment, um, that also lives in the protected layer. artifacts, which are things like memories or any files that you wanna store, that are core to the state. We also keep secrets. and finally, we have this special guardian mechanism that
85. u85 23:00 I'll talk about in a second. Everything else lives up in the evolvable policy layer. So there's tooling, external adapters, the way by which you assemble your contacts. When you have too many skills, how you filter down to the relevant ones. Compaction. So really much of, if you look inside of, like, OpenClaw or, Claude Code's leaked code when it came out, which I did, or Codex, a lot of the logic there is dealing
86. u86 23:23 with this, like, what ends up going in the final message that's sent to the model. All of that should live up in the policy so the model-- so the harness can edit itself at runtime. and finally, I didn't talk about this too much, but we decouple both of these from the environment. So this is another shift that I think is happening now, but people don't fully realize it, or maybe some people do.
87. u87 23:42 You guys might already know. But for the past year and a half, we've thought about agents as, I will SSH into my machine, and I will do Claude dash dash dangerous skip permissions. If I'm really secure, I'll start a Docker container. I'll give it a… And then I'll SSH into that and do Claude dangerous skip permissions.
88. u88 24:00 And this way, everything is self-contained. The agent runs in an environment. The shift that is starting to happen is I think people are looking at agents as running but operating on a separate environment. And so you can have your agent process running on your machine, but spinning up a Docker container where any actions it takes happen on that Docker container. So this is an important distinction because then you could have kind
89. u89 24:21 of one agent that's working in multiple different environments, and the sandboxing happens around the environment where the actions are happening, not around the agent itself. this to me makes much more sense as the general direction I think that we're headed.
90. u90 24:35 Q&A: will the split change as models get more trustworthy?
91. u91 24:35 but- Just a quick- Yeah, go ahead. Abhishek. So, regarding this clip, and perhaps, presumably something from the green box is gonna start moving to the, orange box because we trust the model more.

    

92. u92 24:54 do you think there are some core things that we should never move- Mm-hmm … or is there some sort of, like, understanding of that sort? And in the OS, conversation that happened, I don't know, twenty years ago- Yeah did something like this happen as well? Yeah.
93. u93 25:10 Yeah, that's a great question. Okay, so to repeat the question back, you're basically asking, as our models get better, is this-- wh-- is this split that I present to you model agnostic or will this shift over time? If I were to trust my models more, would I shift this? That's, that's one thing you're asking, right, Abhishek? that's correct.
94. u94 25:26 Yeah. Yeah. That's, I'm getting at. Yeah. Yeah. Um, and the other thing you're asking is, in the OS community, um, how did they deal with this question twenty years ago? Um, I can answer both of those questions. Um, no matter how much you trust your model, unless someone shows me some, like, alignment research that totally changes my thinking, I don't think there's ever a state in the future that I can trust that my model will
95. u95 25:48 never exfiltrate my secrets or will never delete something I don't want. Precisely because when we ask a model to do something, it's taking a limited specification, our words of what we want, and it's expanding it into, like, a very rich space of actions. And just the fact that you're going from a little bit of information to a lot of information means there are assumptions being made in that process.
96. u96 26:07 And alignment to me is making those assumptions line up with what I want as the human. Um, but one, there's just so many different humans that want different things. Like- I, don't believe that alignment is just gonna solve this, where I'll be able to trust-- where I can just tell my model, "Don't ever delete my state," and it won't ever do that.
97. u97 26:25 Um, so for that reason, like I here have intentionally picked what I think is the minimal set, and no matter how smart your model gets, to have the guarantees of safety that I want, you need your state to stay protected and, just managed entirely outside of the model's control. that's my take.
98. u98 26:43 Maybe I'll be proven wrong with time. Um, to answer your question about the operating systems literature, has anyone read the Exokernel paper here? I'm just curious. No? No systems people? Okay, maybe a couple people. Um, there are also, other papers. There's like a microkernel, paper. There is a big debate on this.
99. u99 27:01 Um, if you'll notice, at the end of the day, where did we get to? Um, Exokernel did not win. Our laptops are running the traditional operating system with this kernel. This is why you probably recognize this and not, you know… The one on the right is foreign, I'm telling you about it. Um, the reason that that won out is there is a different trade-off there, where, um, the whole benefit of this design was to give policy flexibility.

    

100. u100 27:24 But it's a lot of work for a application to go and rewrite the file system or the virtual memory or the network stack or the IPC. So in a world where it was a team of engineers that were doing this, where it was humans doing this, the additional flexibility granted by having kind of, this policy up in the user lay-layer just wasn't worth it, wasn't beneficial.
101. u101 27:43 The majority of application developers just wanted to develop their app and leave everything else to the kernel. But in a world where we're now shifting to, it's not humans writing code, we're scaling up hundreds of times more code being written because it's being written by AI, suddenly it becomes worthwhile to expose as much as you can because it's not our effort anymore that costs…
102. u102 28:02 Li-like, we're trying to expose as many levers to squeeze out, performance as we can to customize our systems. And now that this, customization is automatic, I think that this more Exokernel-like model will win out for the harness and agent layer. Does that make sense, Abhishek? yeah.
103. u103 28:18 That broadly, answers the question. Maybe we can, Yeah … have further follow-ups, but we can then, take that later. Yeah. I'm excited to show some results, so I'll, I'll leave time at the end for us to talk more. Um, okay, cool. Um- All right.
104. u104 28:35 An agent as policy + state + sandbox
105. u105 28:35 So two more slides and then I'll get to results, and I'm excited about the results. So, um, now I've kind of presented to you the split that we have in our exo harness and an agent now, if I can define an agent for you, an exo harness is actually a logical construct that spans these three layers. So an agent, an instance of a running agent is a particular set

     

106. u106 28:55 of policy and tooling, plus the conversations history that's stored, the state that's stored in the state layer, plus a particular sandbox. And by extracting out this policy into its own layer, we have an isolated target for safe self-evolution that won't hurt either the conversation history, the state or the sandbox environment.
107. u107 29:14 Um, and we've actually isolated out and can independently roll back or roll forward the conversation history, the environment, or even I don't have an arrow here, but even the policy. If you make a proposal to change a policy and you don't like it, you can roll back afterwards and they're all rolled back independently. Yeah.
108. u108 29:33 Okay.
109. u109 29:34 How self-improvement works: the guardian
110. u110 29:34 So finally, the last slide of the design. To fully collapse the system, if you're curious, how does self-improvement actually work? Once you split these layers up, there's two steps. The first is you have your policy up top. we take the actual code of the policy, and we mount it in the sandbox to be read/write.

     

111. u111 29:53 And so in this way, the policy can make decisions to take actions in the sandbox, and those actions can do any tool calls or whatever general coding it wants, or it can edit the actual policy that's sitting there right on the sandbox. And then in that middle layer, in the Exo harness, I mentioned this guardian mechanism. Whenever the policy wants to swap itself out for a new policy, it'll go and call the guardian, which builds that code.
112. u112 30:15 Actually, builds the code and replaces the running policy with the new policy. but the guardian actually does something else important. It doesn't just replace with the new policy. It keeps both up in parallel to check that the new policy has some safety pr-properties. At a minimum, it could be that your new policy doesn't crash. But a more complex kind of checker, Guardian G, could check that on
113. u113 30:36 some small, set of kind of test tasks that the mo- that the new harness isn't just really dumb. And if that passes, the old policy process is thrown away, and the new policy process is the new running policy. That's it. This is the mechanism by which you can now do self-editing. There's automated rollback, and you need this to be outside of the policy because
114. u114 30:56 otherwise, if the policy dies, you have no way to, to, rep-- fix-- to recover. and we get our self-evolving harness.
115. u115 31:03 Q&A: protecting the guardian, comparison with other harness designs
116. u116 31:03 So quick question. Yeah. Maybe it's a dumb question. No. How do you make sure your guardian function is not modified by your policy model? because, the, policy, the code that's mounted in the sandbox, the editable code doesn't include the guardian. Like, the actual guardian tooling is provided by this protected layer.
117. u117 31:22 The same as there's also a database inside of that Exo harness middle layer, the protected layer that holds conversation history. You literally cannot act on it because it's not in the sandbox. Like it's isolated by actual- Isolated by… Okay. By, architecture. I see. Mm-hmm. Question. So looks like you're doing a rollback when failure happens. Yeah.
118. u118 31:41 So I've seen other systems do this fencing a different way. Like the DeepSeek harness, they do this, spatial temporal compatibility, and you can completely extract, like, the module out. So what's the pros and cons between, like, different design around this? Yeah.
119. u119 31:59 They seemed much more focused on, like… Yeah, they had this, like, multiple components and make changes to certain components and keep certain parts and isolate others. I haven't seen that just to be necessary. I kind of treat this all as one big set of policy, and as the policy evolves, you can roll it back or forward.
120. u120 32:15 I haven't seen the need to kind of split it up into components and per component state. I actually found that DeepSeek harness paper, like, really hard to follow and comprehend. it was, like, very theory. They gave it, like, basically a formal proof- Yeah … their approach would work. Yeah, I didn't find that paper super useful personally. or, so… I can think about this more, maybe at the end, but, I didn't find a big
121. u121 32:37 need for a lot of theory around different components independently evolving and rolling back their individual state. I kind of treat this as one component. Yeah, I find this simpler to reason about and also simpler to code up, but yeah. okay.
122. u122 32:50 Secrets handling and self-editing its own improvement process
123. u123 32:50 So I'll just point out now, this architecture makes it so that edits can't erase your canonical history of snapshots. the policy is only p-replaced with something that passes the guardian, whatever checks the guardian has, G. I, I didn't really talk much about this, but we also keep secrets stored in that middle layer.

     

124. u124 33:07 This is another big problem. People like, you, how do you get Claude to not accidentally exfiltrate your, like, API key, which is sitting in your environment variable or, in your, like… Just we actually partition that. We keep it out of the environment and, intercept any messages as they're leaving the sandbox to swap in the secret so, that the model never sees the secret.
125. u125 33:27 but we get this ability to fully inspect your kind of running log for looking at your own cost, a cache use effect, et cetera, and you're able to learn in the environment, and you have interpreter learnings. So, and the last thing I'll just say, doing it this way, the policy process can edit its own, its own guidance on how to edit itself even.
126. u126 33:51 So if it wants to set up some additional observability to inform its own decisions about how to better change itself in the future, it can do that. Whereas before, when you had an outside agent responsible for kind of editing an inside agent, that outside agent was fixed in its policy, the, like, reviewer agent or whatever. So by having the same agent be responsible for editing it itself, it can also
127. u127 34:10 customize its own policy for how to inspect its behavior to the custom domain it's running in Okay, so, we built this.
128. u128 34:19 Implementation and open source
129. u129 34:19 This thing is real. It's out there. You can go see it on GitHub. I guess here there's one point five K stars. I think it's at two K stars now. so the Exo harness itself, that architecture split up, was about thirty thousand lines of code. Actually, the majority of that comes from the sandbox support. We really wanted the ability to run not just in Docker containers, but off on, in Firecracker, Daytona, E2B, Cell Sprites, et cetera.

     

130. u130 34:40 So you can run locally in multiple containers. If there's not enough kind of space on your machine, you can treat your agent as kind of a control system that farms out to environments that are in the cloud. That's a whole separate side set of benefits. And then Exo itself, kind of the self-improving architecture, ended up being thirteen thousand lines of code and two thousand line-- Rust and two thousand lines of TypeScript.
131. u131 35:01 and, it's running in production now at, at, Braintrust, so they use it for a number of things internally. A-and a handful of other companies also.
132. u132 35:10 Evaluation setup: closed benchmarks and open-world tasks
133. u133 35:10 but for the evaluation, this is what I've been having a lot of fun with the last six weeks. I took a look at two types of environments. So one is a set of closed benchmarks, the typical way we evaluate things. Um, I picked kind of continual learning bench. there's suite tasks for coding, App rules for general tasks. SREGym is an infrastructure set of benchmarks.
134. u134 35:31 And for each of these, I compared kind of a self-improving version of Exo with a non-self-improving version, meaning without the ability to edit its own code. And after an initial period of seeing, twenty or fifty tasks, it varies based on the benchmark, I let them then both kind of run out on new tasks and report their performance.
135. u135 35:51 The other one that I'm really excited about, is open world tasks. So with a high-level goal, open world, operating on the order of hours in a dynamic environment. Has anyone here played RuneScape? A couple people have played RuneScape? Okay. Others haven't played RuneScape? Have people heard of RuneScape? Very fun game.
136. u136 36:07 It was one of the key games from my childhood. and I'm gonna actually show you a bunch of results from Exo playing RuneScape out with other players dynamically. and in particular, I'm actually gonna show you and compare it to Codex and Pi, which I think are really fair comparisons because they're kind of state-of-the-art agents that we work with today.
137. u137 36:26 so I'm gonna start mostly talking about the open world a bit. I'm gonna spend maybe five, six minutes on the remainder of this talk, and I'll leave ten-ish, twelve minutes for us to have discussion.
138. u138 36:35 Q&A: how does the policy actually evolve?
139. u139 36:35 Sorry, can I? Yeah. Can you talk a little bit about the actual learning and evolving mechanism for the policy layer? Mm-hmm. Is it some sort of, Bayesian optimization or gradient descent or what- No. How do you- No. if I had more time… A different way for me to talk about this would have been coming from all of the AI-driven discovery research, which if you're familiar

     

140. u140 36:55 with AlphaEvolve or OpenEvolve or Sky Discover or even, GEPA, a handful of these things, the way they, work really simply. They just say, "Here's a system, here's an evaluator, and here's an LLM. Run the evaluator on the system, take the logs, give it to the LLM, and ask it to improve it, improve the system." No Bayesian blah, blah.
141. u141 37:18 It's remarkably simple, but actually, if you let the model, especially… And here's the thing, we have to be thinking about our systems not for where models are today, but for where the trends of where models of where they're going. And if you believe these models are going to get better and better at making decisions, showing them the design of your system and asking it, "What's the way to improve this?" actually is an increasingly higher return approach.
142. u142 37:40 Makes sense. Yeah. Thank you. Yeah.
143. u143 37:43 RuneScape results: Exo vs. Codex and Pi
144. u144 37:43 Okay. So, um, I set up Exo with access to this, OSRS SDK, which gives you like an MCP interface by which to go and call and take actions in the game. and what's kinda cool here is this is actually on a real open public server. i-it's a huge map.
145. u145 38:00 There's many different skills to level up. There's many different actions to take. and all of Exo, Pi and Codex all kind of got access to the same tooling. and I gave it three very high-level tasks. One was a combat objective, which is to take a brand-new account and train it up to maximum combat to maximize its st-stats on combat.
146. u146 38:20 the other was to obtain as much gold as possible, as much money in the game within a fixed amount of time. and the third was to obtain an item that's kind of relatively hard to make. It takes many hours to kinda get there. Um, and all this was run with GPT 6.1 Sol and what I see is that Exo consistently, very strongly outperforms Codex and Pi on their efficiency.
147. u147 38:43 So all of the tasks across all three were completed successfully on multiple runs. The difference was largely in how much time and how much money it costs to actually get to the completion of the task. But what's more interesting to me is actually I have this, Other way 'round. Yeah, so here, like this is the overall view. At the time of finishing the task, they all finished the tasks at
148. u148 39:01 approximately similar times, but Exo was thirty-one to sixty-two percent cheaper on these open world tasks. What's really interesting to me is if we just zoom into one of these, the combat task, if you look at these curves, it doesn't start off cheaper. This doesn't come for free. So if I kind of zoom in on this early time, the very first thirty minutes of gameplay or so, we can see that Exo is actually more expensive than
149. u149 39:22 both Codex and Pi as it starts out. But if you look and look at the logs, if you actually read what it's doing, it'll go and experiment with different techniques in the game and then build tools or changes to its own policy mechanisms, each of those is marked with dots, that pretty quickly flatten out the curve. and I can give you some examples of what these are in a second.
150. u150 39:40 Like one example is, for maybe a deep legal task, if you're working on like some legal processing, there's a bunch of documents, and you need to keep all of that in your context to, remember what you're working on as you're referencing many, many documents. Whereas in RuneScape, it's actually fine for you to keep a much shorter context. You probably only need the last couple of steps of actions because it doesn't really
151. u151 39:59 matter what you did twenty minutes ago. The policy by which you assemble that context is fixed in Codex. Codex has a set of rules with a maximum context window of a million tokens, and when it reaches nine hundred thousand tokens, it'll go and call compact. And so indeed, you see, you can go even look over here on the decision time on the right.
152. u152 40:17 You can see Pi and both Codex also have these sharp jumps up and down as the context window is filling up, and then it compresses and goes back down. Very early on, Exo realizes that it doesn't really need longer history, and so it goes and edits its own internals of how it assembles context to keep only the last couple messages, which keeps its context very short. I, know that there I have decision time, but, the decision time is
153. u153 40:39 directly proportional to how long the context is when it's fed into the call, so that's why I make this claim. but you can see that you-- the cost is drastically lower for Exo to kind of reach its final goal, even though the progress looks about the same across the three agents. Okay.
154. u154 40:56 then the other-- Yeah, please. So, yeah, so I'm wondering what's the context management technique that the Exo comes from. Did you give it to this? yeah. It would after… It would set itself a goal for the next, up to two hundred turns, and then every two hundred turns it would summarize… First of all, within each two hundred turns, it would summarize-- it would only keep, I think,
155. u155 41:18 the last like ten steps that it took. And then after… A-and it would keep around some memory note, a file where it would keep track of what it's doing. And then every turn it would summarize that down to like a single sentence of what it tried in that turn, and then it would keep a running log of that. I see. Yeah.
156. u156 41:34 Q&A: cost-awareness and the meta prompt
157. u157 41:34 Did you specifically instruct the model to be cost-aware? Because normally when all these coding agents do stuff, they don't care about cost. I, yeah. I gave identical prompts to all three. which, for all… It was not, explicitly… I asked it to be efficient. I did not… Yeah.

     

158. u158 41:52 That-that's all I asked it, actually. Yeah. That's it. Yeah. Great. Yeah. a related question. Yeah. The meta prompt, which is I guess like the prompt that you're asking the model to improve itself or whatever- Mm-hmm. Is that aware of, these sorts of objectives, or is it sort of very general purpose, say, improve yourself in whatever way you see fit?
159. u159 42:12 Yeah. Yeah, it's, it's the latter. it's general purpose. So the meta prompt, if you look inside the initial starting policy, is more along the lines of like, you're an agent that can edit itself. This is what your code layout looks like. If you wanna go look at it, here's a file that describes it. it doesn't say anything RuneScape specific or even efficiency specific. Thank you.
160. u160 42:31 Yeah. Okay.
161. u161 42:35 Second run: carrying learnings to a new account
162. u162 42:35 here's the other thing I'm excited about. Once you've done this once, you can go and take the same exact harness you just asked to speci-- you just specialized, and go ask it on a brand-new account in a new… So at a new point in time, new set of players, go work on the same exact task. And I did this for all three, and to be fair, I let Codex and Pi keep their entire history.

     

163. u163 42:53 So same session for Codex and Pi. so same across all three. I just gave them a brand-new account and said, "Go try this again from scratch." and we can see that, like, for some tasks, it's about the same actually, and it's because for, like, earning coins, getting money, Exo pretty quickly found the optimal strategy for its abilities.
164. u164 43:11 And so there wasn't that much time where it was floundering around. There wasn't that much time to learn until it got to the right, strategy. But for combat, for example, a lot of time earlier on in the first run, it was learning how to optimize its own ability to do combat. And so in the second run, it was getting twice as much, experience per, dollar, meaning it was twice as efficient at kind of, leveling
165. u165 43:32 up, as it was in the first run. And so the gap actually kind of, grows even, particularly for tasks where there's things to kind of learn more, where there's more repetition. okay. oh yeah, and I show this here. You can kinda see what happens.
166. u166 43:48 Again, it's, to me this is super interesting, but like, stays very flat and there isn't even a period where it's like, worse than the other two. And actually, in the second run, it, didn't edit tools at all. It already had all the tools it needed and the context and the policy that worked well for combat training. I have all the good results here.
167. u167 44:07 Limitations: where self-improvement doesn't pay off
168. u168 44:07 as a good researcher, I should tell you, it's not all good. in particular, there are classes of tasks that are not very well-suited to learning repetitive policy edits. and in RuneScape in particular, those are, doing quests. So if you're doing something where it's like there's actually no pattern across what you're doing, which in these games is like, you know, for a

     

169. u169 44:27 particular quest, you have to go talk to this person, then go collect some resource, then do this other thing, and it's very specific per quest. There is an overhead to trying to build tools to help you do your thing, and if those tools aren't gonna be used in the future, this is actually-- this hurts you. So this doesn't come for free. It doesn't beat on everything. But for the classes of kind of work where there is repetition, which
170. u170 44:46 I think is actually a lot of the work we're going to try to be… Like, as we bring these agents into our kind of office work, there's a lot of repetition what we do. So for classes of work like processing insurance claims or, monitoring your infrastructure, checking rollouts, like there are certain patterns of things we do that if we can let the harness specialize, we can really drive down its cost.
171. u171 45:07 yeah.
172. u172 45:09 Q&A: "dreaming," reward sparsity, and benchmark gaps
173. u173 45:09 Kind of reminds me of, like, neuroplasticity to some extent, where it's like you repeat certain things and then your neural connections become stronger, and that's how, like, humans can become better- Mm-hmm …if they do, and they can generalize for different tasks. what I was curious about was if you also study, like, memories on, like, dreaming in addition to this, because I'd imagine, like, if you were to give,
174. u174 45:29 like a Exo harness the ability to just, like, look at its past reasoning, look at its past actions, and dream and reflect and then self-improve- Mm … would that be needing to change the harness? Yeah. How does that compare to the Exo study? Th-this is a great question. So the question is on, if I can repeat it back, the, like, concept of dreaming is you at runtime just do whatever you're going to do, but you have some
175. u175 45:49 downtime when you can look back on your past traces and think about what would I change to do differently if I were in that scenario again based on the trace. Yes, you will get benefit from that. I didn't kind of compare it to that 'cause that would be me implementing, like, yet another solution. I just let Codex run by default, and I'm choosing that as state-of-the-art. My response is that what Exo-- what I watched Exo do was that when you're
176. u176 46:08 doing dreaming, you are stuck reflecting on a set of traces that you watched. You've basically done a rollout, and then you reason about the rollout. Whereas Exo is aware of the fact that it has the ability to edit itself, and as a result, if you actually look at the traces, it intentionally decides, "Let me try doing this thing so that I can have some logs of this thing to look at." Mm. Like, that is a key difference.
177. u177 46:29 Mm. Otherwise, you just do whatever you think is the best in that moment. You don't have the notion of I will actively push in this direction to measure it to decide whether I should do that. Yeah. Okay. yeah. So I know you mentioned about like the repetitiveness of the thing, so you can better, create a working like harness change.
178. u178 46:49 What about like the reward design that actually in your like, evolving loop? Do you need to be very dense or actually some sparseness is okay? Because I'm not super familiar with the game, so. Yeah, okay. This is a great question. So, we're talking about reward density and like reward sparseness. Yeah, yeah. in many like real world tasks, like a proof, y- it's a very sparse reward.
179. u179 47:09 Like you know at the end when the thing is proven, you don't know the two weeks on the way there. But we're seeing that agents are able to kind of intelligently guide themselves towards the final results by reasoning about how decent does this proof look or how close to c-- I know it's not right, but is it close, or am I feeling like I'm making progress? So I'm designing this explicitly for tasks that are, You're right.
180. u180 47:30 One reason this is a little bit easier in RuneScape is there's experience that you can measure. Yeah. So you have some kind of intermediate, guidance. what I will say is I have here some results also on, other benchmarks that don't really have an intermediate like, reward that are like, SREGym is one here, which is like, we bring up a- an environment, infra environment,

     

181. u181 47:52 that's literally running your apps and your, like services in Kubernetes. We inject a fault somewhere, and we ask an agent to figure out what the fault was. No intermediate reward, just at the end it has to fix the fault. And I see the same kind of wins in those environments too. That's cool. So I haven't done a deeper study on sp- sparsity or, or, richness of reward. But obviously where there's a richer reward, it's easier to self-optimize.
182. u182 48:13 But I'm seeing the same things hold for less, Less dense … dense. Yeah. okay. Let me just… There's a bunch of other numbers here. More or less, I see this hold across different benchmarks. This does not hold for tasks that aren't related. So when I like ran this on TerminalBench, for example, I don't see the same benefits because it's all in the terminal, but the tasks for
183. u183 48:33 TerminalBench are all so different. One of them is predict the next chess move and the other is like, like, I don't know, train a model that predicts the blah, blah, blah. They're such different tasks that there isn't a benefit across them. There's actually a cost to reflection, so it's more expensive to run something like Exo on those. I just-- The gap I see and the thing that I was hoping you guys would affirm is there seems to be a huge gap between the benchmarks people are using to
184. u184 48:55 say that their models are good and the classes of actual work that we do. Because the work we do, I don't do work like T- TerminalBench at all. Like, I don't have a hundred unrelated tasks. I have one agent that works across a series of tasks that are actually pretty related. So to me, the reason that these gains are available and they're kind of low-hanging fruit is because the way we're thinking about evaluating what
185. u185 49:15 is working well and what isn't is not aligned with how the real world works. That is kind of where I'm coming from. Okay.
186. u186 49:24 Personalized harnesses and policy "branches"
187. u187 49:24 all right. I- Do you envision a world where like everyone can have their own personalized harness that optimize towards- What I've been testing Yeah. I suspect that there will be profiles of harnesses. 'cause I think I work on… You might wanna… I work on a few different projects at once, and I actually might wanna have a harness

     

188. u188 49:43 specialized to the particular kind of work I'm doing in that project. I also, like, sometimes use agents to help me manage my, like, health data, and other times I'm using them for coding, and other times I'm, like, planning a date. And those are very different profiles. Presumably, you can create a set of- Totally. Totally … like the harness should be smart enough to create a set of harnesses.
189. u189 50:01 Yes. Yes. And to me, that is kind of a, tree of almost branches. if you think of the policy as like a Git repo, you, could have branches that are kinda specialized to certain areas.
190. u190 50:12 Designing systems aligned with the bitter lesson
191. u191 50:12 all right. I, do have one more slide I really wanted to save because this is maybe the overarching theme of what I wanna advocate for. As we're doing systems research, we have to be moving towards systems research that is aligned with the bitter lesson. And what I mean by this is we are constantly adding, in systems research, I think, surrounding stuff on top of the latest model.
192. u192 50:33 People tried, they would had like a core model that was doing something, and they were like, "Let's add an external model that's a advisor model that then occasionally says something to the inner model to help it." And then what happens? Three months later, a new model comes out that's better, and it just swallows up all of that additional stuff we built over top. All of those papers from nine months ago that were like, "Let's just have a
193. u193 50:53 multi-agent system," they are useless now with the latest model because the latest model just does that thinking on its own. And so rather, the case I'd like to make is, we have to be doing work where when the model gets better, the benefits of the surrounding system get better as well. They have to grow proportionally.
194. u194 51:11 And so the goal is to design our systems for intelligence, where better models make the system actually more useful. And to do this, you have to intentionally reprioritize your design goals of your systems because things like flexibility, modifi- modifiability, and legibility become first-class design goals of your systems. Whereas before, they were just kinda nice to have.
195. u195 51:30 Because this is what exposes the system to evolution to the better model itself. how to get there is the question. Exo is one possible answer. I have another slide, but actually, I will just stop, there. So, if you wanna come check this out, the code is up, on, GitHub at, github.com/exoharness/exo.
196. u196 51:48 We have kind of forked a little bit. Exo is now… There's a self-improving side, but it's also a mo- a, harness, an agent runtime. And so when you go and see the code, you'll notice that. but, yeah. I'd be very happy to take questions. Thank you all for listening.