The Unit Economics of AI: What It Really Takes to Deploy LLMs at Scale | Ido Rosen of Amazon SDO

The source, passage by passage

1. u1 0:00 Ido's day-to-day at Amazon SDO
2. u2 0:00 I'm excited to speak with Ido Rosen about his work applying AI across Amazon. In this conversation, we cover everything from deploying the smallest LLMs in the real world to the unit economics of frontier models in the AI market. Tune in for this fascinating conversation.

   

3. u3 0:21 I'm very excited today to speak to Ido Rosen from Amazon. And I think you've got probably one of the most interesting jobs that I've come across. It really spans across very practical things, but also very science-y things. So I really want to dig into that today. First of all, what does your day-to-day look like? I spend most of my time investigating interesting science and engineering
4. u4 0:43 and a third to a half of it working with teams across stores and ads and a couple of other organizations, covering mostly the practical aspects of deploying both agents and other types of machine learning and AI models into the wild. I would say it's a good mix of of coding,
5. u5 1:04 investigating, and spending a lot of time interacting with other scientists.
6. u6 1:09 Agents everywhere, or one feature at a time?
7. u7 1:09 So this is perfect, because I think there’s a lot of chatter about AGI and agents, and we see all of the benchmarks climbing and so on. And I think you've got a really front-seat view to see how we actually deploy these things. So the first thing I want to learn from you is when you see adoption of AI, do you see this as, we just

   

8. u8 1:32 have a lot of agents just doing a bunch of things, or is it much more kind of feature at a time deployment? What’s your experience? I think the deepest penetration in terms of AI or machine learning in industry has already happened. So much of the internet that you interact with is driven by ML models, whether they're embedding models or even something like collaborative filtering or
9. u9 1:58 other forms of AI like these LLM chatbots or agents. I don't think we realize just how much of our lives is touched by models like this, including more recently this kind of convergence of reuse of transformer models for all kinds of other use cases that don't have a chat interface.
10. u10 2:19 The ambient customer experiences that you wouldn't think of as AI experiences are probably the ones that I'm most impressed by right now. As far as deploying agents into the wild, surprisingly little of our lives really involve waiting for LLMs to think – today the amount of seconds of my day that are spent waiting for an LLM
11. u11 2:43 to think are probably a good metric for how deeply agents have penetrated into the life of your average person using their computer.
12. u12 2:50 Why we rarely wait on LLMs (yet)
13. u13 2:50 Yeah, I think that's an interesting metric, because definitely there is a big difference between, I think, what we see on the very frontier when we see agents running for days and days and days and the day-to-day experience of an average person. So what do you think...is there like a fundamental blocker there?
14. u14 3:09 Or do you think the models are not very good? Or it just takes time? What's your view? Companies and engineering teams do whatever they can to hide the less pleasant parts of using agents. In the example of agentic recommendations, like product recommendations, for example, you can do a lot of pre-computation ahead of time.
15. u15 3:30 It can get very expensive if you don't know which customer is coming to the website next. And by the way, I should say I'm speaking generally. I'm not speaking about any particular technology that that Amazon uses. There's the simple approach of just caching, or precomputing. And
16. u16 3:48 there are more complicated options, like there are techniques you can use to get sequence models, whether they're LLMs or other kinds of sequence transaction models, to retain a long horizon theme across a lot of output without necessarily doing the same kind of inference time scaling that, you know, “reasoning” does for the chatbots that we interact with.

    

17. u17 4:10 Every one of these use cases, or every one of these experiences, customer experiences, they live on some simplex of quality, cost, and latency, and they have to make trade-offs in in that space. When they trade off, for example, quality for latency, that has to be a very intentional trade-off. It affects the product experience in a very serious way.
18. u18 4:31 As an example, if you searched a shopping website and every time you typed a search, it took 10 seconds to load, that would probably turn off many users. There are certain experiences where you definitely see lots of LLM usage, but you don't see the latency added in the same
19. u19 4:51 way that you do from coding agents that you're probably used to waiting around to do inductive thinking.
20. u20 4:57 Is there a product or business-model gap?
21. u21 4:57 So do you think there is any space? Do you think it's kind of just our ways of designing products? Is it about, I don't know, business models? Or what is it? Because it sounds like LLMs could do that, right? So in my day-to-day life,

    

22. u22 5:14 if I'm looking for something really important, I could have an agent spend an hour, two hours looking for something, and I go back and forth. And then, I don't know how much I spend in inference on that, but it could be $100, right? Clearly for any business, it's impossible to do that for all customers. But is there an innovation gap on the product side or business model side
23. u23 5:36 where we can actually deploy these kinds of agents? Much like in the mid-noughties when GPUs started getting better, and the machine learning models started getting bigger, I think these are two paths that intersect. As some of this research continues on making models more efficient,
24. u24 5:56 whether that's through quantization or other methods, these models move on that simplex so that you can achieve more reasoning, higher quality results with lower latency. And cost and latency are pretty closely related. At the moment, cost is mainly compute time.
25. u25 6:13 So I'm not sure if you can look at those as independent directions, but where I'm going here is that the customer experiences tend to drive which tools you'll use or which models you'll reach for or which base models you'll start from. And the production kind of constraints – maybe I have a latency budget of a hundred milliseconds for a given task – will
26. u26 6:36 severely limit what a real deployment looks like and will cause people to have to get very very creative. More directly relevant to your question is that it's really surprising how many teams that I talk to don't consider unit economics up front.
27. u27 6:53 Unit economics: the head, torso & tail of a catalog
28. u28 6:53 Here's an example. Suppose that you have a shopping website with a very large catalog. In that case I would assume that there's a head, a torso, and a tail of the of the sales distribution of any catalog. You can probably spend a lot more on the head of the distribution, the stuff that sells the most,
29. u29 7:13 than you can on the stuff that doesn't sell any. And in the extreme, if you have a product on a website that sells zero units, then return on investment for any kind of LLM inference is approximately zero, because the numerator is zero.
30. u30 7:32 At some point you have to average these into your unit economics and your ROI, and the unit economics of inference are (for the moment) reasonably punishing if you have a low margin business, but quite reasonable for a high margin business. If you look across companies I think that's why you see agents being deployed into spaces where
31. u31 7:53 there's a reasonably high margin, either because I'm replacing some very high cost process or because I'm going to cause a high margin business to grow. I think for most of these use cases, it just boils down to what the unit economics are – what can I expect? What is my return on tokens? Basically. If you're talking about the standard models available, as opposed to being able to fine-tune
32. u32 8:18 or modify the architecture of some of these open weight models.
33. u33 8:21 Are small models getting better?
34. u34 8:21 Yeah, that's actually very interesting to get into as well. But I’d be interested to get your view of the trends in that lower and cheaper model. Do you see models get substantially better at the cheap end? Or do you see the models getting cheaper and cheaper?
35. u35 8:42 There's discourse on the top end and how close we are to AGI and all of these things. But do you actually see that? Do you feel it in your day-to-day work? Benefits kind of trickling down on these kind of cheaper, quicker models. I do, yeah. It's kind of funny to call a 1.5 or a 4 billion parameter model a small model. I think about five years ago I would not have said that.
36. u36 9:07 I definitely think that this particular structure of foundation model, whether they're the open weights base models or other internal models, I do think there's been a significant improvement, but I think that improvement comes from fairly boring places. I think it comes from a combination
37. u37 9:26 of better representation and finding interesting engineering hacks and ways to make more of the compute that you spend on those models, even the relatively low parameter count of these models do more work for you. I've already mentioned quantization is one dimension in which you can scale
38. u38 9:46 up the actual useful work per flop or per watt. There are a few other dimensions that you can scale into to get better utilization of the joules, I guess, you're spending. And I think it it does eventually boil down to useful compute per unit energy, not these kind of intermediate measures like tokens or others.
39. u39 10:06 What actually works: post-training, fine-tuning & hacks
40. u40 10:06 And for people maybe who are building these kinds of fast or very large scale systems, can you talk about any of the hacks or best practices you've seen, experienced? What really makes a difference in your experience? Well, okay, so there's what makes a difference in terms of being able to use a bigger model or use a more expressive model.
41. u41 10:29 and then there's what makes a difference in terms of performance on the actual task. I wouldn't be surprised if most use cases out there are using these kind of narrowly fine-tuned – and when I say fine-tuned, I really mean post-trained, like regardless of whether you're doing SFT, RL, or something else. But these narrow models that have been optimized
42. u42 10:51 for the specific task...we have a paper from back in the days of BERT that one of the scientists I get to work with, a fellow named Viktor Chernozhukov, wrote on hedonic pricing using transformer models. This is ancient history now, but you at the time you could start from BERT and you could add a few layers, and you could end up with a model that you did some combination
43. u43 11:11 of CPT and additional training to turn it into a price prediction model. Similarly, the moment that you're in a different output domain than language... You're now working on a machine learning problem rather than an AI problem and now it's just a question of well, does adopting one of these base models, and continuing from there, net
44. u44 11:31 me any kind of benefit over starting from scratch. But I think you're you're asking a different question. I think you're asking what techniques for adapting LLMs to real use cases are people using. I think unsurprisingly they're the same techniques that people use to adapt LLMs to LLM use cases.
45. u45 11:51 So things like RL post training in gyms, and SFT. There are some people playing around with various ablations and I'm probably not qualified to enumerate all the different ways in which people are messing with these models’ architectures.
46. u46 12:08 Team skills and "simplify and invent"
47. u47 12:08 You know, I'm interested. You mentioned quite sophisticated techniques, because whenever I speak to builders, especially it could be big companies, small companies, quite often it turns up, “We did some prompt engineering.” “We maybe at best built some evals or maybe eyeballed some outputs.” That kind of level. And maybe the rest is,

    

48. u48 12:29 I don't know, software engineering. That's certainly my experience as well. So how do you think about your team structure? Team structure capability or skill sets that you think make a difference in your world. What sort of things do you value right now? So first of all, I do think, for the majority of the typical use cases that you'd reach for an LLM, whether that's helping me with a coding task or summarizing a document
49. u49 12:56 and doing those things at scale, you're absolutely right. You don't really need to do much to the existing set of small and large models to get them to work on those well understood use cases. I think when you start to, for example, use an LLM to drive a browser, and you realize that, wait,
50. u50 13:17 I need to drive this browser at scale and I need to do it with spending less than one penny per website. Or you encounter some new constraints of your problem space that don't exist for the document summarization use case, for example, then you have to innovate and that's where these kind of more sophisticated approaches come in or that's where you know you're left
51. u51 13:39 with no choice but to add complexity. There's this leadership principle at Amazon, “invent and simplify”. And a few of us have suggested that it really should be “simplify and invent”. Because for most of the the use cases, like I said, the methods are are quite boring.
52. u52 13:58 They're things like LLM as judge, evals on a benchmark data set.
53. u53 14:03 Detour: evals and why LLM-as-judge can mislead
54. u54 14:03 I'm sorry, I'm not answering your question here, but we should take a brief detour to evals just because you happen to work at Arena. I'd love that. One thing that I'm just surprised by is that there aren't more open source frameworks out there that allow you to perform some of these model selection or model customization tasks using

    

55. u55 14:25 the sort of end-to-end business reward function. So for example, did this chat help or or harm the customer experience, as measured by whether the customer bought more stuff afterwards or less stuff, as measured by units sold or by GMS or other ways that that you would measure a traditional change to a website
56. u56 14:47 through for example an A/B testing framework. This is kind of related to this disappointment that I've had around realizing that actually for many quantitative tasks, LLM as judge methods can be quite misleading. I'm curious, why do you say LLM as a judge would be misleading
57. u57 15:06 for quantitative tasks? I think it's pretty commonly known now that LLM as judge methods tend to have self-preferential bias. And in this setting, you know, I'm potentially using an LLM to evaluate which model should be used. For example, if that's the decision variable,
58. u58 15:24 if it's model selection. Or which which prompt is better if it's prompt optimization. Whichever judge LLM you choose has its own set of biases, and I think the self preferential bias is more an artifact of the training distribution now increasingly including synthetically generated text from that same LLM or its its predecessor.
59. u59 15:46 Setting that aside for a minute, in each case I'm choosing a biased judge. And what I really want is, especially in adversarial settings or in settings where there is a ground truth, I really want to get at that adversarial process or that that ground truth. Stock markets are the best example that I can think
60. u60 16:06 of of an adversarial setting here. If I have an LLM giving me an investment strategy, It's kind of obvious that I don't want to use LLM as judge there. I want to simulate my trading strategy and have a single scalar: what was my PnL at the end of a given time period using these different models. In surprisingly many settings, with a little bit
61. u61 16:24 of extra work there does exist some kind of either ground truth or or objective measurement and for the ones that there doesn't, I'm not sure that it matters. So for the ones that there doesn't exist such a measurement, maybe LLM as judge or random number generator are equally valid.
62. u62 16:44 Lessons from a decade in quant trading
63. u63 16:44 Going back to to to evaluation for a minute, I spent a decade in in the quant trading industry before coming to Amazon. In that industry, I had the most objective manager that I've ever had in my career, the futures market. And I knew at the end of the day, because we were doing fairly short term trading,
64. u64 17:03 high frequency trading, I knew okay, I'm helping or I'm hurting the the company with my work. My trading strategy starts working better, or not. And I would know by the end of the day. There's one single PnL number and it's your performance evaluation. I'm sorry. It's also hopefully proportional to your pay in that industry.
65. u65 17:23 And similarly in simulation where you simulate against past market data I would be able to reasonably predict when things would go wrong with a given new trading strategy, which a trading strategy here, for all intents and purposes, is just a new machine learning model. And
66. u66 17:43 Arena, auctions & choosing the best model
67. u67 17:43 Arena is actually one of the first, I think, adversarial settings to kind of test models, the sort of pairwise tournaments that Arena ran, which really fascinated me, and it made me wonder whether, just like in HFT, whether there are structural advantages to different ways of
68. u68 18:07 constructing your auction scoring function for your tournament. And it turns out that's true in many more settings than than I would expect. As an example, in an ads business you have an auction scoring function as well, determining which of the many ad creatives you could display in a widget. You will go display to this customer for this given request in real time
69. u69 18:32 is not just a simple “pick the creative that offered the highest bid price” – there's a scoring function because you have to consider relevance and all kinds of other factors. Similarly, if you could construct your LLM usage as some kind of setting like that where, okay, I have many choices,
70. u70 18:56 many different, potentially, models that I that are bidding for this response, and many potentially different fine-tunes or other versions of that model. I mean you could go really big here, but just concretely, suppose I have a coding project and I want
71. u71 19:16 to bid it across the various coding agents, right? If I could express that auction scoring function or that objective function for that task cleanly, in a way that, you know, it would be hard to reward hack, and that I couldn't take advantage of some kind of cheap structural features of that market, then I would have a much more grounded
72. u72 19:41 way of choosing the best model. And today the the way that we do that is for most engineers, I think, in the industry, this is not not in any specific company, it's that we go look at Arena and we go look at artificial analysis and then we pick a favorite and we download the desktop app for that one. And then we go use that.
73. u73 20:02 But I think the future is gonna be much more multi, multipolar and multiplexed across these models. Like I think labs are going to have to specialize at some point. And so back to evaluation. I just wish more of our more use cases out there could express their auction scoring function, or could express their their objective function
74. u74 20:24 for their real world use case, even if it's post hoc, even if it's after you've made the choice. Because if it's after you've made the choice, you can still look at it like a causal inference, you can still do some kind of causal analysis there to see, okay, is there enough signal in in the response for me to know whether I could have made a better choice.
75. u75 20:43 The skills gap in deploying LLMs
76. u76 20:43 I think you're right. I get the sense that there's definitely space for innovation, but we haven't quite grown the muscle to do it. I think there's just probably not been enough cycles of teams like yours, actually skilled enough and focused enough to do that kind of thing. And quite often, from my experience of doing this kind of thing
77. u77 21:02 in terms of deploying LLMs, it's normally a couple of people that are probably engineers. They are used to a very deterministic world where you write code and it runs. And I think it becomes fuzzy. So we need to probably have some new skills and techniques to do that kind of thing. I think you're right,
78. u78 21:21 there's not really enough open source frameworks. There's not enough approaches to do that. But it's interesting how you're talking about lab specializing and we kind of touched on open source.
79. u79 21:34 Open source vs frontier models
80. u80 21:34 I’m very curious to hear, what do you think about the industry, how it's shaping up? Do you feel like open source is going to take over and just spread into all of these use cases? Or do feel like there's still a lot of space for more frontier models? Where do you land? Actually I think this is a really good question.
81. u81 21:52 I don't have a crystal ball and I don't know how this how this industry will evolve. I do feel like a lot of the marketing from these labs does not consider what happens under competition.
82. u82 22:04 Inference premiums & the labs' free cash flow
83. u83 22:04 For example, today some labs are able to extract a fairly large inference premium. That's the difference in price per token and their compute cost per token, which I don't know what their actual compute cost per token is, but I can compare the price that I pay across different coding agents and different models and different labs.
84. u84 22:26 And without even naming any of them, because I think this is equally true of the inference platforms and not just the model provider sort of labs, the free cash flow of the lab today with their current business models of essentially selling tokens... (I'm oversimplifying because they have subscription
85. u85 22:44 pricing. They have all kinds of other, they have per seat licensing – various labs have various different business models and pricing schemes) But let's take the simple one of of selling tokens through an API. The unit economics of that are pretty straightforward, right? [(Price - compute cost) * quantity] is pretty much my inference margin, if that's the only or the primary way that the
86. u86 23:08 the labs make money, then that's their free cash flow. As a slight detour because I used a fancy word there, “free cash flow” – free cash flow and reinvesting that free cash flow in your customer experience is how Amazon is still growing 30 years on, in my opinion, right? This is all my personal opinion.
87. u87 23:26 I really like that that company because it somehow managed to stay in this growth stage. Every company has a business life cycle of startup growth, maturity, cash cow, death – or sorry, acquisition. Companies that are in the startup phase, They typically need to fundraise,

    

88. u88 23:47 not always, but typically they need to raise money in order to fuel their next customer growth. Companies in the growth stage or the maturity stage often can fuel their their growth in terms of customers, in terms of business lines, etc., from their long-term free cash flow.
89. u89 24:05 The alternative, by the way, in the profit taking stage or the maturity stage is to do something else with that free cash flow such as issue dividends or do other fun financial things with that money. So if you look at the labs as businesses, which you know a lot of people have,
90. u90 24:23 and you can make bear cases and bull cases for the labs with these. I take no stance on on either side. I'm excited to see where they go next. But if you look at their their free cash flow equation, if it's that simple we sell tokens business model,
91. u91 24:40 What if open weights stay 6 months behind?
92. u92 24:40 then assuming that the open weights models stay a matter of months behind the labs, and all this is subject to their licensing models, et cetera, but let's say that we have a permissively licensed open weights model that is you know three to six months behind the state of the art.
93. u93 25:03 on any given day. And maybe with this past week's events, maybe it's six to nine months for for now. But let's say that the it averages around six months. Then does that mean that every use case that the frontier model would have been able to charge that large inference premium on, six months later
94. u94 25:22 is essentially just paying their compute costs because the the alternative to a negotiated agreement with the lab is to just go self-host an open weights model? I think that's a real hard business question. At some point, the frontier models and the open weights models six months behind them will have consumed a good chunk of the
95. u95 25:46 economically valuable work in the world. and I'm not saying that we're at that point. I'm saying that maybe at some point in the future they will have. And at that point, as you're approaching even like a double digit percentage of that point, does the customer of these things decide to decide to just wait six months and pay significantly less? Or does
96. u96 26:09 the frontier lab's ability to set a reasonably high price change? Do they lose that pricing power because their customers have very, very reasonable alternatives? I think these are great questions. I have opinions about what the answers probably are, only from seeing them in other industries,
97. u97 26:28 but there might be something special about this industry.
98. u98 26:31 Does intelligence plateau? A lesson from nature
99. u99 26:31 Related to that, I think there are these competing bets being made. One bet is that the value of intelligence goes up exponentially, and another bet is that it plateaus. If it plateaus, the open weights models eventually catch up. If it plateaus, then it starts getting harder and harder to produce more intelligence for whatever that means in a given use case.
100. u100 26:52 And if it's exponential for the foreseeable future, then maybe that those inference premiums continue to rise. I think different labs have taken different bets on this. Let me borrow an example from nature. I'm sorry, you you've got me riled up thinking about this particular problem. We live in an environment...
101. u101 27:10 At some point we lived in a much scarier environment that involved, you know, tigers and and things chasing after us and maybe eating us. Today we live in an environment, just like any agent, that requires a certain amount of intelligence to survive in. I need to know where to get food and and water. I need to know enough things to put
102. u102 27:29 food on the table and to pay for rent or a house. Human brains probably haven't evolved for this exact use case, but brains have evolved in animals and in humans over millions of years to solve these kinds of reasoning problems, right? And yet you don't see brain complexity exponentiating.
103. u103 27:54 This is a stupid philosophical point. So please don't take the metaphor too far. If in the environment that you live in it is evolutionarily beneficial for your brain to grow in complexity, then it should, right? And, you know, I recognize that evolution happens over much, much slower timescales here but especially with some of these
104. u104 28:13 new auto research environments I would think that we would see stronger incentives to to make models even larger and to make them even more complex. And what I mean by complexity is the complexity of the of the model itself, not necessarily the complexity of the algorithms. It can't all just be SGD with extra steps,
105. u105 28:35 right? There has to be more to it than that, I hope. But I don't think we do see that in nature. There is some kind of optimal efficiency trade-off for for brain size. I suspect that
106. u106 28:48 Pareto frontiers and why labs will specialize
107. u107 28:48 in these business settings, you're gonna encounter a similar sort of optimal trade-offs or some kind of Pareto frontier – – Pareto frontiers abound across these different trade-offs. Usually what happens in a setting like that, where you have a multi-objective optimization and multiple different frontiers, is that players specialize. They might specialize in a few different ways.
108. u108 29:11 One way is for example, you might have different environments in which they they compete. So we've talked about coding a lot, but there's also search – I think you have a search Arena as well. And there's also, you know, video generation and all these other places where you have diffusion transformers or other similar architectures
109. u109 29:32 you can explore. And in each one of these there is this bear chasing the frontier labs, which is that there are other labs that will just take that large fixed capital investment of pre-training and release it for free in the form of open weights models.
110. u110 29:56 And so in each one of these you have this same challenge of we don't yet know if intelligence plateaus, but let's suppose that we don't need that answer yet. If the open weights models get licensed in such a way that they can be used to replace commercially all of these other frontier
111. u111 30:21 LLMs over time, then that probably has implications for the business cycles and the investment strategies in these these frontier models. Assuming that there are still returns to scaling.
112. u112 30:38 If instead there are only returns to research, right? If instead of scaling, the next frontier is complexity, is research to find some new model architecture that solves problems or some new loss function that makes learning more computationally efficient, which must exist, then this idea can go out the window.
113. u113 31:00 Then basically we're playing potentially a lottery between various research organizations to see which one finds the next set of incremental improvements that lead to a step function improvement in the overall model performance.
114. u114 31:17 A "spiky" world of superhuman niches
115. u115 31:17 I think this is an interesting point – I think you're right, like we're not in a paradigm that there's some kind of general learning and the LLMs clearly are not learning as efficiently as humans. So if that carries on for some time, it really feels like we're not going to get this kind of
116. u116 31:36 super general super intelligence across everything. But because LLMs are so spiky, we can take advantage of that. And there will probably be a few spikes which have such extreme return on investment. It's hard to judge how good of an example it is, but what struck me about OpenAI's microchip announcement is
117. u117 31:58 the way the position is like, AI helped us design it, or optimize it or whatever it is. I don't know how true this is, but even if it's not exactly true in the way we are imagining, it still feels like there will be some spikes where if you just put billions and billions into that spike,
118. u118 32:18 which has extreme return, then it will be such hyper super intelligence in that spike. And if it does give you a return, then it feels like surely why not throw more and more money into it? So it feels like maybe we will be in this kind of crazy spiky world where LLMs can't write a well-written essay, which kind of feels like they still can't.
119. u119 32:41 And yet they can design super microchips that no human could ever design. What do you feel about that? Do you feel like that spike story could happen? I think it could, I think it's already happened in a few places. Like for example, clearly there were returns to scaling for for coding and for tool use.
120. u120 33:03 There were also returns to scaling things like the number of RL gyms or or environments that you expose these models to. There are all of these different treadmills that that the labs are on for things like data collection. I'm looking for longer and longer horizon,
121. u121 33:22 more interesting and computationally, you know, challenging for the LLMs, use cases out there and for each one of those, I want to construct a gym. That's kind of a typical post-training task for a generalist model. If companies are willing to spend inordinately large sums on these kinds
122. u122 33:42 of expeditions, then clearly they're expecting there to be some return on that investment. I think a a better way to
123. u123 33:49 Asymmetric returns: radiology to recommendations
124. u124 33:49 to frame this is just in the return on inference in different industries. Not only is it jagged but it's asymmetric in returns, right? There are there are many industries for which a false positive or a false negative is much more harmful than others. Radiology is a good example,
125. u125 34:10 right? A false negative in an LLM that reads an MRI image is probably not a good thing for the patient. A less consequential example is recommendation systems, where you see LLMs getting used increasingly. I agree with you that there might be industries where there
126. u126 34:32 are returns, but those returns are usually capped at some point. As an example, if you've spent all of your disposable income you're not gonna buy more stuff probably – maybe that's not yet proven because people borrow on credit cards all the time, but if you spend more than your disposable income regularly, it's
127. u127 34:52 you're gonna go into debt and eventually you'll run out of ability to loan money. And at some point there's a point of exhaustion for that industry. We don't really understand that for most of these industries. Certainly we don't understand it for software engineering. i'm a firm believer that there will be more demand for software engineers. When you change an industry, by deploying an LLM into it for example,
128. u128 35:14 you're also changing the size of the industry potentially. And so to your point, knowing which of those use cases, which of those industries deploying agents into actually causes the pie to grow versus just being a cost savings, is probably one of
129. u129 35:34 the key differentiators. Back to your earlier question about predicting how deploying LLMs will change that industry...
130. u130 35:39 How big can the AI market really get?
131. u131 35:39 As far as the future of the labs go, I think you can go back to thinking about unit economics. I don't think you need to imagine these spikes in the frontier, I think you can you can take for any given industry what the addressable market is and you can reason about what would happen under competition if more than
132. u132 35:59 one lab enters the race, in that use case or in that market. And you very quickly get back to, well, which ones of these are going to be competition to the bottom on price, and the ones that are, you you end up coming up really with much, much smaller expected values
133. u133 36:20 for their real kind of addressable market. I'm kind of dancing around this idea that maybe the marketing that they're putting out there is overestimating the the actual addressable market from the current form of of LLMs or of the way that they're deploying AI. And have only done so insofar as they haven't necessarily modeled market impact
134. u134 36:41 and competition when they produce that marketing. I mean it's in their interest to to market these things as multi-trillion dollar industries.
135. u135 36:50 Wrap-up
136. u136 36:50 You’re so right. I think this is probably the biggest question we've got right now in the industry, is how big is the market going to get? Because all of this investment is like, you have to justify it at some point. And I think unit economics, ultimately what it comes back to. I think this is a very good place to close.
137. u137 37:07 I really enjoyed our conversation. We went all the way from deploying smallest models in the practical use cases, the super fast inference and all the way to how big the market could get. So this was a really fascinating conversation. Thanks a lot for joining us at Arena.
138. u138 37:26 Yeah, thank you. This has been a lot of fun to think about. I'm very excited that Arena exists because it's one of the few places where we actually get to see models under competition in real time. So thank you.