AI will change how the world creates software. As I wrote two weeks ago, the SaaSpocalypse wasn’t caused by one model release.

The pace of change is accelerating faster than even the forecasters can track.

~4% of all public GitHub commits are now authored by Claude Code, a 42,896x increase in 13 months, projected to hit 20%+ by year-end of 2026.

There are teams where software engineers have a mandate to write no code. They direct AI, monitor AI, review AI, and design the systems that AI executes.

Yet, the core of being a good software engineer hasn’t changed.

System design, functional requirements, latency trade-offs vs. the flexibility of the system. These are all about having the right taste and the right judgment.

AI transforms how engineers execute. It frees them to focus on the higher leverage work that matters: deciding what to build and why.

Trust: why companies buy software

Organizations are teams of people. And people are lazy, in the best sense. They want to do the things they want to do.

If a small business, a real estate brokerage, or a restaurant is focused on delivering a product or service that is not software, they want someone else to manage the software. They want someone else to worry about it. And that someone is better as an entity you can hold accountable than one you can’t.

A company can have 500 different SaaS tools handling 500 specific tasks. They want a button that does a very specific job, backed by a vendor they can call when it breaks.

For example, for lawyers and consultants, folks may tell you lawyers are out of a job because AI can write and review contracts. Consultants are done because AI can produce better slide decks, deeper research, and sharper deliverables.

But ultimately, lawyers and consultants are in the people business.

Partners drive business through relationships. Relationships are built on trust.

People hire law firms to mitigate and transfer legal risk. People hire consultants to bolster their own reputation; or to transfer reputational risk to the consultancy.

For SaaS, companies buy software they trust so they can focus on their own core competence.

People use software so they can focus on doing what people do best.

The bull and bear case for SaaS companies

The bull case is software companies become leaner and enormously more productive from AI.

The best SaaS companies will get stronger and dominate the market, the middle and the bottom will fizzle out. The internet cycle that impacted newspapers and media will repeat itself for software. The star companies will build at 10-100x the speed. They will build better products, more personalized, more fit for purpose. Salesforce already cut support headcount from 9,000 to 5,000 while combined Agentforce and Data 360 ARR approached $1.4B at 114% year-over-year growth.

Not only that, but the barrier to entry will also be increasingly low. Market competition will greatly increase, for the benefit of companies and consumers.

The bear case is software becomes entirely undifferentiated.

Rapid diffusion of software capabilities makes software engineering ubiquitous to any person with a smartphone. Goldman Sachs draws comparisons to newspapers — an industry where share prices declined 95% between 2002 and 2009. The internet cycle that impacted librarians and newspaper clippers will repeat itself.

We should remember why companies buy software.

Both cases are plausible, I’m 100% in the bullish camp.

On February 5th, OpenAI and Anthropic dropped GPT-5.3 Codex and Opus 4.6 on the same day. The SaaSpocalypse wiped $550B from software stocks. Hyperscalers committed $660B in capex for 2026.

Something big IS happening.

But if you look closer, the progress and change is driven by something small.

Something small is happening.

The March of the 9s

A knowledge worker makes 50-100 micro-decisions to complete a single task.

1. Open the right folder (find the right email)

2. Search the right term in Google

3. Click the right result, read it, go back to search page

4. Save the source, write a note on the research

5. Draft an intro paragraph to the research brief

6. etc...

Sequential, contextual, compounding. That is how human knowledge work manifests today.

Here’s the math that matters

Each additional “9” is a small absolute gain at step level reliability. Going from 99.5% to 99.95% is less than half a percentage point.

But over 50 sequential steps, that half-point is the difference between ~78% and ~98% success. Over 500 steps, ~half a day of autonomous work, it’s the difference between total failure and actual reliability.

This is why AI seems to “suddenly” get good. It crossed a compounding threshold.

Code generation by LLMs has existed since GPT-3 Davinci and the original Codex models in 2021. But “vibe coding” didn’t take off until Karpathy coined the term in February 2025. Again, it’s a story of compounding progress.

The story of computing

The $660B in hyperscaler capex is the bill for the next 9, and the next 9 after that. Each 9 requires more compute, more data, more energy. The returns diminish at the step level but compound at the system level.

As I wrote last October:

Power unlocks compute. Compute unlocks AI. AI unlocks intelligence.

Satya Nadella’s framing is instructive:

[…] the company just literally provisions a computing resource for an AI agent, and that is working fully autonomously. That fully autonomous agent will have essentially an embodied set of those same tools that are available to it. So this AI tool that comes in also has not just a raw computer, because it’s going to be more token-efficient to use tools to get stuff done.

But an always-on agent running 500-5000 steps needs more 9s of reliability to work.

GPT-5.3 Codex and Opus 4.6 didn’t shock because they were categorically different.

They shocked because they added another 9.

And at this point on the curve, each 9 is a phase change.

Investors were suddenly questioning: had Chinese AI labs caught up, were moats were illusory, and all that CapEX might be wasted?

A year later, AI adoption is exploding, costs are dropping as all labs push the Pareto frontier (best efficiency for best performance).

And if you’re building with AI, your expectations from three months ago (in November 2025) are probably already outdated.

AI Adoption Explodes

Back in June 2025, I wrote that ~1 in 10 (~10%) U.S. businesses used AI. Last month in December 2025, the US Census Bureau measured AI usage at ~18% of businesses. Modest improvement?

Ramp’s credit card data tells a different story: ~47% of businesses are now actively spending on AI tools.

But the real signal is from AI lab’s revenues. Epoch AI surveyed 421 forecasters at the start of 2025. For 2025, they predicted $16B in combined annualized revenue for OpenAI, Anthropic, and xAI.

The actual number? $30.4B.

Compared to the end of 2024, OpenAI and Anthropic had about ~$6.4M annual revenue run rate, $30.4B is a ~4.8x annual revenue in 1 year!

It keeps getting cheaper too: open and closed source models

Now the supply side. Every lab is pushing the Pareto frontier, better models at lower cost, faster than anyone expected.

Near the end of 2025, Google’s Gemini 3 model series has retaken the Pareto frontier.

How did it happen?

DeepSeek showed the way. When I analyzed their inference economics last March, frontier AI labs were running 80%+ gross margins on their API business. DeepSeek’s pricing was near break-even, everyone else was (or is still) printing money.

But DeepSeek did something else: they open-sourced their inference infra techniques and concepts.

Now, open source inference software (like vLLM) are integrating these learnings directly, diffusing the learnings to anyone with access to GPUs (not solely to labs with billions of capital).

HackerNews folks ran some back of the napkin math on the cost economics (of the vLLM release) coming with ~$0.30/1M output tokens at ~2k tokens/second (very very fast) running DeepSeek’s latest model.

Compared to DeepSeek’s public API price of $0.42/1M output token and Gemini 3 Flash’s $3/1M output token, it is possible for even small and medium sized enterprises can in-source their own LLM inference and beat 1st party pricing.

DeepSeek’s innovations didn’t just help DeepSeek. They raised the floor for everyone.

Claude Code psychosis… just keep building

Demand doubled versus forecasts. Costs continue to plummet. The Pareto frontier moves measurably in 3-6 months.

For builders, you have to reset your expectations every few weeks. Claude Code’s Opus 4.5 integration took the AI world by storm in December.

The model you’re building on will be obsolete in ~6 months, it is a feature. Not a bug.

I’ll leave others to debate whether there’s a valuation bubble or the benefits of one.

Today’s goal isn’t value capture, it’s value creation. The labs are competing on who can make intelligence cheaper, faster, more accessible.

For founders, this is the best possible environment. The tools get better every month. The costs keep dropping. The adoption keeps accelerating.

My tip is to follow smol.ai’s AINews email newsletter, aggregates signal from the noise every day.

Embrace the Claude Code psychosis?

TSMC’s dilemma, OpenAI’s hyperscaler dreams, and ambient computing

Last week, OpenAI and Broadcom confirmed long-suspected rumors of a custom AI chip (rumored since 2023). OpenAI will buy 10GW of custom AI chips from Broadcom over the next 4 years, starting in the second half of 2026.

OpenAI will be effectively triple-sourcing its long term AI chip supply: NVIDIA, AMD, and Broadcom.

Remember…, NVIDIA, AMD, and Broadcom purchase the vast majority of their AI accelerator chips from TSMC.

TSMC’s dilemma

If OpenAI needs 1 or 2 orders of magnitude more chips, from 2GW (today’s) to 20GW (over the next 5 years) to 250GW (in 10 years), where are the new fabs?

As of last week’s earnings, TSMC has not yet announced any increase to its rate of capital expenditures since the OpenAI deals.

Tricky business for TSMC going forward. They face a few strategic dilemma moving forward:

• Managing supply distribution to top customers: In the short term, TSMC continues to have an effective monopoly on leading-edge chip fabrication.

  • Now with one end-user (OpenAI) exerting demand pressure via 3 vendors (NVIDIA, AMD, and Broadcom):

    • How can TSMC effectively price and manage supply allocations to its top customers?

    • Will this start a bidding war?

• Taiwan vs. U.S. domestic production: TSMC and Samsung (TSMC’s closest competitor) both have operating (or close-to-operating) chip fabs in the U.S.

  • Yet, TSMC continues to call out U.S. fabs are more expensive1 to operate, diluting gross margins by 200 to 400 basis points (~2-4%).

  • How will TSMC manage CapEx discipline overseas vs. Taiwan in the face of more competitive pricing environment?

  • Will TSMC be stuck investing in the U.S. for geopolitical reasons, and thus giving up gross margins in the long term?

• Intentional undersupply or risk management?: C.C. Wei (TSMC CEO & Chairman) recently said:

  I believe we are just in the early stage of the AI application. So very hard to make right forecast at this moment.

  • For each 1GW increase in AI data center buildout, TSMC estimates it costs its customers ~$50 billion in investments.

  • Yet, for 2025, TSMC’s annual CapEx guidance is ~$42 billion. If inline with comments and historic trends, its 2026 CapEx will be closer to ~$42-50 billion2.

  • OpenAI alone expects to build out 20GW of capacity over the next 4 years, effectively investing ~$1 trillion in total.

  • Does TSMC believe the ramp in AI demand, and will it show up in its CapEx guidance next year? Or is it intentionally undersupplying as a risk-management strategy?

OpenAI’s hyperscaler dreams via vertical integration

Broadcom is the chip supplier for Google’s TPUs, Meta’s custom AI chips (MTIA), and Apple’s not yet released AI server chips.

Custom chips for cloud workloads are not new. Cloud computing hyperscalers—Amazon Web Services, Microsoft Azure, and Google Cloud Platform—all deploy custom chips in their cloud services. Custom chips are used mostly for CPUs workloads, but increasingly for GPU workloads.

Effectively, OpenAI chose to pursue the same vertical integration strategy as the cloud computing giants. It is trying to become a hyperscaler.

But will Oracle be the new hyperscaler instead?

For hyperscalers, Amazon, Google, Microsoft, and META all build their custom chips to specifications for maximum efficiency. The chip designs take into account the data center’s design, configuration, and operations.

Ultimately, cloud computing and hyperscalers are economies of scale businesses. Managing this long-lead time and capital intensive business requires showing its compute investments have increasing returns to scale—meaning for each dollar of revenue and investment, its returns should grow.

But, OpenAI’s cloud computing (including GPU compute) is primarily served via Microsoft Azure and Oracle’s cloud services3. They run mostly everything on Azure.

Oracle and Azure will no doubt accrue economies of scale savings, as they expand and operate their data centers with diligence. But they have no mandate to pass on such savings to OpenAI.

Even if AI demand explodes as planned, OpenAI could be holding the bag and kneecapping their long term strategic position.

Instead, Oracle and Larry Ellison may emerge as the new hyperscaler and capturing a stranglehold in the AI computing value chain—just like they did in the database business three decades ago.

Always on agents: ambient computing will need more compute

In OpenAI’s Broadcom announcement podcast (the full transcript is reproduced at the end of this post), Greg Brockman revealed some of his thinking on agents over the longer term:

Our intent is to turn ChatGPT into something that helps you achieve your goals. The thing is, we can only release [ChatGPT Pulse] to the pro tier because that’s the amount of compute that we have available. Ideally, everyone would have an agent that’s running for them 24/7 behind the scenes, helping them achieve their goals.

Ideally, everyone has their own accelerator, has their own compute power that’s just running constantly.

That means with 10 billion humans, we are nowhere near being able to build 10 billion chips.

Having an ambient AI agent running all the time seems to be a no-brainer. But yet, there are not nearly enough compute resources to be a reality or cost-effective.

For an ambient agent to be productive, it needs to observe, listen, and react in real-time.

Today, the closest offering to a realtime AI mode is OpenAI’s realtime API feature.

• It is also one of OpenAI’s most expensive API offerings, nearly ~6x output pricing.

• For audio, it costs $32/1M input tokens and $64/1M output tokens–that is compared to GPT-5’s $1.25/1M input and $10/1M output.

The high price reflects the high amount of resources required to serve an AI model in realtime (with very low latency).

As I’ve written before, when running AI models, there is a trade-off between latency (speed for the model to respond) and throughput (how many responses can be processed simultaneously).

• Having higher throughput is more efficient, thus less costly for a model provider.

• Faster responses (low latency) requires very low batch sizes (very low throughput).

• Physically, that means the model service provider is holding GPU chips idle (or below optimal utilization) in order to guarantee a very timely response.

Lastly, when combining the need for low latency (realtime) with thinking (reasoning mode), the compute requirements for ambient AI grow exponentially.

• Today’s OpenAI realtime model has no reasoning function, partially for this reason.

• Reasoning will linearly increase the context of the conversation, which will exponentially increase the compute requirements for the model to continue to respond to the user.

Here, I make a prediction that always-on agents (or ambient computing) is the missing UX (user experience) component in today’s AI paradigm. But the technology is bottlenecked by the availability and cost of compute, the effects can only slowly diffuse over time.

Once realtime agents can be served as cheaply as Google search (marginal cost of near zero for Google), then ambient computing can diffuse quickly across the internet.

The following is a transcript of an OpenAI Podcast episode.

The host’s questions are in bold. The other speakers are:

• SA = Sam Altman (OpenAI)

• GB = Greg Brockman (OpenAI)

• HT = Hock Tan (Broadcom)

• CK = Charlie Kawwas (Broadcom)

The transcript is lightly edited for content and clarity.

What are we talking about today? What brought you all together?

SA: Today we’re announcing a partnership between Broadcom and OpenAI. We’ve been working together for about the last 18 months designing a new custom chip.

More recently, we’ve also started working on a whole custom system; these things have gotten so complex you need the whole thing.

We will be starting in late next year, deploying 10 gigawatts of these racks of these systems and our chip, which is a gigantic amount of computing infrastructure to serve the needs of the world to use advanced intelligence.

So this is going to entail both compute and chip design and scaling out?

SA: This is a full system. We closely collaborated for a while on designing a chip that is specific for our workloads.

When it became clear to us just how much inference capacity the world was going to need, we began to think about whether we could do a chip that was meant just for that kind of a very specific workload. Broadcom is the best partner in the world for that, obviously.

Then to our great surprise—this was not the way we started—as we realized that we were going to need the whole system together to support this as this got more and more complex, it turns out Broadcom is also incredible at helping design systems. So we are working together on that entire package, and this will help us even further increase the amount of capacity we can offer for our services.

Hock, how did this come about? When did you first talk about working together on this?

HT: Other than the fact that Sam and Greg are great people to work with, it’s a natural fit because OpenAI has been doing and continues to do the most advanced frontier models in generative AI out there.

As part of it, you continue to need compute capacity—the best, latest compute capacity as you progress in a roadmap towards a better and better frontier model and towards super intelligence.

Compute is a key part, and that comes with semiconductors and, as Sam indicated, more than semiconductors. We are, even though I say it myself, probably the best semiconductor company out there. AI is a very exciting opportunity for us; my engineers are pushing the innovation envelope on newer and newer generations of semiconductor technology. For us, collaborating with the best generative AI company out there is a natural fit.

Vertical Integration and Scale

And this isn’t just chips, it’s going out to scale—like 10 gigawatts. I have trouble kind of even understanding that. What does that even mean when you’re talking about 10 gigawatts?

SA: The vertical integration point is really important. We are able to think from etching the transistors all the way up to the token that comes out when you ask ChatGPT a question and design the whole system: all of the stuff about the chip, the way we design these racks, the networking between them, how the algorithms that we’re using fit the inference chip itself—a lot of other stuff all the way to the end product. By being able to optimize across that entire stack, we can get huge efficiency gains, and that will lead to much better performance—faster models, cheaper models, all of that.

As you get that better performance and cheaper and smarter models, one thing that we have consistently seen is people just want to use way more. We used to think we’ll optimize things by 10x and we’ll solve all of our problems, but you optimize by 10x and there’s 20x more demand. So 10 incremental gigawatts—this is all on top of what we’re already doing with other partners and all the other data centers and silicon partnerships we’ve done. Ten gigawatts is a gigantic amount of capacity. And yet, if we do as good of a job as we hope, even though it’s vastly more than the world has today, we expect that with very high-quality intelligence delivered very fast and at very low price, the world will absorb it super fast and find incredible new things to use it for.

The hope is that the kinds of things people are doing now with this compute—writing code, automating more of enterprises, generating videos in Sora, whatever it is—they will be able to do that much more of it and with much smarter models.

AI Designing AI Chips

Greg and Charlie, when you think about historically when people have tried to develop chips or hardware to suit whatever was the current mode for using computing, what examples have you looked upon historically to figure out how to plan forward? What’s been inspiring you?

GB: The number one thing, honestly, is working with good partners. It’s very clear that we as a company are not able to do everything ourselves. Getting into actually building our own chips for our own specific workloads was not something we could do from a total standstill without working with Hock and Charlie and Broadcom. It’s been really incredible to lean on their expertise, together with our understanding of the workload.

It’s been very interesting to see the places where OpenAI is able to do things very differently from the rest of the industry or the way that things would historically be done.

For example, we’ve been able to apply our own models to designing this chip, which has been really cool. We’ve been able to pull in the schedule; we’ve been able to get massive area reductions. You take components that humans have already optimized and just pour compute into it, and the model comes up with its own optimizations.

We’re at the point now where I don’t think any of the optimizations we have are ones that human designers couldn’t have come up with. Usually our experts take a look at it later and say, “Yeah, this was on my list, but it was like 20 things that would have taken them another month to get to”. It’s really interesting that we were coming up on a deadline working with Charlie’s team and we were running optimizations. We had a choice: do we actually take a look at what those optimizations were, or do we just keep going until the deadline and then take a look after? We decided, of course, you got to just keep going.

We’ve really been building up this expertise in-house to understand this domain, and that’s something we think can help lift up the whole industry. I think we are heading to a world where AI intelligence is able to help humanity make new breakthroughs that just would not be possible otherwise, and we’re going to need just as much compute as possible to power that.

One very concrete example is that we are in a world now where ChatGPT is changing from something that you talk to interactively to something that can go do work for you behind the scenes.

If you’ve used features like Pulse, you wake up every morning and it has some really interesting things that are related to what you’re interested in; it’s very personalized.

Our intent is to turn ChatGPT into something that helps you achieve your goals. The thing is, we can only release this to the pro tier because that’s the amount of compute that we have available. Ideally, everyone would have an agent that’s running for them 24/7 behind the scenes, helping them achieve their goals.

Ideally, everyone has their own accelerator, has their own compute power that’s just running constantly.

That means with 10 billion humans, we are nowhere near being able to build 10 billion chips.

There’s a long way to go before we are able to saturate not just the demand, but what humanity really deserves.

CK: For us, it’s been absolutely exciting and refreshing because the beauty of the work we do together is the focus on a certain workload.

We started by first looking at the IP and AI accelerator, which is what we call the XPU. Then we realized very quickly that we can now go from the workload all the way down to the transistor. As Greg was just explaining, we can both work together to customize that platform for your workload, resulting in the best platform in the world. Then we realized, as Sam was saying, it’s not just that XPU or accelerator; it’s the networking that needs to scale it up, scale it out, and scale it across.

Suddenly we saw that we can actually drive the next level of standardization and openness that not only benefits us, but I think will benefit the entire ecosystem and get Gen AI to an AGI much faster. So, very excited about the technical capabilities of the teams we have, but also the vision and the speed at which we’ve been moving.

The Scale of the Mission

I’m still kind of wrapping my head around the scale of it. This is a global effort. What comparisons have you been able to draw for this to other examples in history?

SA: I always think the historical analogies are tough, but as far as I know... I don’t know what fraction of global GDP building the Great Wall was at the time, but in a lot of ways that you would look at the AI infrastructure build-out right now, you would say it’s the biggest joint industrial project in human history. This requires a lot of companies, a lot of countries, a lot of industries to come together. A lot of stuff has to happen at the same time, and we’ve all got to invest together.

At this point, given everything we see coming on the research front and all of the value we see being created on the business front, I think the whole industry has decided this is a very good bet to take. It is huge. You go to one of these one-gigawatt data centers, and you look at the scale of what’s happening there—it’s like a tiny city. It’s a big, complex thing.

GB: To the point of this being a massive collaborative project, whenever I call Charlie, he’s in a different part of the world trying to secure capacity, trying to find a way to help us build what we’re trying to do together.

CK: One of the coolest things I was thinking about is what we’re doing together in this wonderful partnership: we’re defining civilization’s next-generation operating system. And we’re doing it at the transistor level, building new fabs, new manufacturing sites, all the way to building these racks and ultimately the 10 gigawatts of data centers you’re talking about.

It’s an important thing to keep track of. Often people get fixated just on the chips themselves, and it’s kind of like thinking the National Highway Project was about selling asphalt, or railroads are about steel. In reality, it’s the things that become possible on top of that.

HT: This is like railroads, the internet. That’s what I think this is becoming over time: critical infrastructure or a critical utility. And more than just critical utility for, say, 10,000 enterprises; this is a critical utility over time for 8 billion people globally. It’s like the industrial revolution of a different sort. But it cannot be done with just one party, or we like to think it can be done with two, but it needs a lot of partnerships; it needs collaboration across an ecosystem. Because of that, it’s important to create, much as we say about developing chips for specific workloads and applications, standards that are open and more transparent for all to use, because you need to build up a whole infrastructure to become a critical utility for 6 billion people in the world. We’re very excited, which is why we think we make great partners, because I think we share the same conviction.

It is about scaling computing to create breakthroughs in super intelligence and models. It’s building the foundation of that.

The Drive for Custom Silicon

You guys have a lot on your plate. Why design chips now?

GB: This project, we’ve probably been working on it for 18 months now, and it’s moved incredibly quickly. We’ve hired some really amazing people. We found that we have a deep understanding of the workload, and we work with a number of parties across the ecosystem.

There’s a number of chips out there that I think are really incredible, and there’s a niche for each one. We’ve really been looking for specific workloads that we feel are underserved and how can we build something that will be able to accelerate what’s possible.

That ability to say that we are able to do the full vertical integration for something we see coming, but it’s hard for us to work through other partners—that’s a very clear use case for this kind of project.

HT: Yes, and more than that. Computing is a big part of what’s gating this journey towards super intelligence, towards creating better and better frontier models. A lot of it comes down to computing—and not just any computing, but computing that is effective, high performance, and efficient, especially on power. What Greg is saying is exactly what we learned and saw here.

For instance, if you want to train, you design chips that are much stronger in computing capacity, measured in TFLOPs, as well as network, because it’s not just one chip that makes it happen; it’s a cluster.

But if you want to do inference, you put in more memory and memory access relative to compute. So you are, over time, creating chips optimized for particular workloads and applications as we go along. That, at the end of the day, is what will create the most effective models on a platform that you want to create end-to-end.

GB: And also, one piece of historical context is that when we started OpenAI, we didn’t really have that much of a focus on compute. We felt that the path to AGI is really about ideas, it’s about tryouts and stuff.

Eventually, we’ll put the right conceptual pieces in place, and then AGI. About two years in, in 2017, we found that we were getting the best results out of scale. It wasn’t something we set out to prove; it was something we discovered empirically because everything else didn’t work nearly as well. The first results were scaling up our reinforcement learning in the context of the video game Dota 2.

GB: Did you guys pay attention to the Dota 2 project back in the day? It was a super cool project.

We really saw if you scale it by 2x, suddenly your agent is 2x better. It’s like, okay, we have to push this to the limit. At that point, we started paying attention to the whole ecosystem. There were all sorts of chip startups with novel approaches that were very different from GPUs. We started giving them a ton of feedback saying, here’s where we think things are going; it needs to be models of this shape. Honestly, a lot of them just didn’t listen to us. It’s very frustrating to be in this position where you say, “We see the direction the future should be going,” but we have no ability to really influence it besides trying to influence other people’s roadmaps.

By being able to take some of this in-house, we feel like we are able to actually realize that vision, and again, in a way that we hope we can show a direction and other people will fill in. Because the amount of compute required to bring our vision of AGI to the world—10 gigawatts is not enough. That is a drop in the bucket compared to where we need to go.

SA: It’s a big drop...

Looking Ahead

What becomes possible with this when you’re building your own chips for inference and for training? Where can you take this?

SA: To zoom out a little bit, if you simplify what we do in this whole process to: melt sand, run energy through it, and get intelligence out the other end...

What we want is the most intelligence we can get out of each unit of energy, because that will become the gate at some point. I hope what this whole process will show us—from the model we design to the chip to the rack—is that we will be able to wring out so much more intelligence per watt. Then everybody that’s using these models in all of these incredible ways will do so much with it.

HT: And you control your own destiny. If you do your own chips, you control your destiny.

It’s interesting to think about how the things that we’re doing today are pretty amazing, but we’re using stuff that wasn’t actually designed specifically for the way we’re doing it.

SA: The GPUs of today are incredible things. I’m very grateful, and we will continue to need a lot of those. The flexibility and the ability to let us do fast research is amazing. But you are right that as we get more confident in what the shape of the future is going to look like, a very optimized system to the workload will let us wring more out per watt.

CK: And it’s a long journey that takes decades. If you go back to Hock’s example, take railroads, it took about a century to roll it out as a critical infrastructure. The internet took about 30 years.

This is not going to take five years; it’s going to take a long time. As we collectively, especially with this partnership, continue to figure out ways to wring out more tokens, we’ll discover that for this training or research, maybe a GPU is great, or maybe we can take what we’re doing with Greg. It’s actually a platform that allows you, like a Lego block, to take things in and out. Suddenly we can get another XPU or an accelerator for next-gen that’s targeted at training or inference or research.

GB: To the point that Sam said of GPUs have really come an incredible way, in 2017 when we started looking at all these other accelerators, it was very non-obvious what the landscape would look like in 5 or 10 years. I think it’s really a testament to companies like NVIDIA and AMD for how much the GPU has just moved forward and continued to be the dominant accelerator. But at the same time, there’s a massive design space out there, and what we see is workloads that are not served through existing platforms. That’s where that full vertical integration is something unique.

SA: The first cluster OpenAI had, the first one that I can remember the energy size for, was 2 megawatts. We got things done with those two.

I don’t remember when we got to 20; I remember when we got to 200. We will finish this year a little bit over 2 gigawatts, and these recent partnerships will take us close to 30.

The world has done far more than I thought they were going to do. It turns out you can serve 10% of the world’s population with ChatGPT and do the research and do Sora and do our API and a few other things on 2 gigawatts. But think about how much more the world would like to do than they get to do right now.

If we had 30 gigawatts today with today’s quality of models, I think you would still saturate that relatively quickly in terms of what people would do, especially with the lower cost we’ll be able to do with this.

But the thing we have learned again and again is, let’s say we can push GPT-6 to feel like 30 IQ points past GPT-5—something big. And that it can work on problems not for a few hours, but for a few days, weeks, months, whatever. While we do that, we bring the cost per token down.

The amount of economic value and surplus demand that happens each time we’ve been able to do that goes up a crazy amount. To pick a well-known example, when ChatGPT could write a little bit of code, people actually used it for that. They would very painfully paste in their code and wait and say, “Do this for me,” and paste it back in. Models couldn’t do much, but they could do a few things. The models got better, the UX got better, and now we have Codex. Codex is growing unbelievably fast and can now do a few hours of work at a higher level of capability. When that’s possible, the demand increase is crazy. Maybe the next version of Codex can do a few days of work at the level of one of the best engineers you know—or maybe that takes a few more versions, whatever, it’ll get there. Think how much demand there will be just for that, and then do it for every knowledge work industry.

GB: One way I like to think of it is that intelligence is the fundamental driver of economic growth, of increasing the standard of living for everyone. What we’re doing with AI is actually bringing more intelligence and amplifying the intelligence of everyone. As these models get better, everyone’s going to become more productive, and the output of what is possible is going to be totally different from what exists today.

And is that a motivating factor for you, the fact that every time you create these new efficiencies, it just benefits so many more people?

HT: From our side on hardware and compute capacity, where the rubber hits the road on this, it’s really incumbent on us to keep optimizing, pushing the envelope on leading-edge technology. There’s still room to go, even from where we are as we go from two nanometers forward, and even smaller than two nanometers, as we start doing all kinds of different technology. It is really great, exciting times, especially for the hardware and the semiconductor industry.

SA: What Broadcom has done here is really quite incredible. It used to be extremely difficult for a company like ours to think about making a competitive chip; in fact, so hard we just wouldn’t have done it. I think a lot of other companies wouldn’t have done it as well. This customized chip and system to a workload just wouldn’t be a thing in the world. The fact that they have pushed so hard and so well on making it so that a company can partner with them and they can do a miracle of a technology chip quickly and at scale... unfortunately they do it for all of our competitors too, but hopefully our chip will be the best. It’s really quite incredible.

GB: And not just what they can do for us today, but looking at the upcoming roadmap, it’s so exciting the kinds of technologies that they’re going to be able to bring to bear for us to be able to utilize.

HT: It’s just the excitement of enabling and collaboratively building models—ChatGPT-5, 6, 7, on and on. Each of them will require a different chip, a better chip, a more developed, advanced chip that we haven’t even begun to figure out how to get to, but we will.

CK: We’re actually looking forward to that because my software engineers now already use that from a software point of view, and it’s delivering efficiencies of dozens of engineers. On the hardware side, we’re not there yet.

But with respect to compute, when we started building these XPUs, you can build a maximum certain number of compute units in 800 square millimeters. Today, we’re working together to ship multiple of these in a two-dimensional space. The next thing we’re talking about is stacking these into the same chip, so now we’re going in the Y or Z dimension. Then the last step we’re also talking about is bringing optics into this, which is what we just announced: 100 terabits of switching with optics integrated all into the same chip.

These are the technologies that will take compute, the size of the cluster, the total performance and wattage of the cluster to a whole new level. I think it will keep doubling at least every six to 12 months.

What kind of timeframe are we talking about? When are we going to first start to see what’s coming out of the relationship?

SA: End of next year, and then we’ll deploy very rapidly over the next three years.

CK: Greg and I are talking about this at least once a week. We just had a chat earlier today on this.

GB: We’re really excited to get silicon back starting very soon, actually. My view of this whole project is it’s not easy. It’s easy to just say, “Oh, yeah, 10 gigawatts,” but when you look at what is required to actually design a whole new chip and deliver this at scale, get the whole thing working end to end, it’s an astronomical amount of work. I would say that we’re very serious. Our mission is to ensure that AGI benefits all of humanity, and we’re very serious about “benefits everyone”. We really want this to be a technology that is accessible to the whole world, that lifts up everyone. You can really see that in trying to make the world be one of compute abundance, because by default, we’re heading towards one that is quite compute scarce.

My wife feels it when she’s trying to get more Sora credits; it feels very scarce.

GB: We feel it so concretely. Teams within OpenAI, their output is a direct function of how much compute they get. The intensity on who gets the compute allocation is so extreme. What we really want is to be in a world where if you have an idea, you want to create, you want to build something, you have the compute power behind you to make it happen.

Gentlemen, thank you very much for sharing this with us. It’s going to be very exciting to see where this goes, and I hope we can keep talking about this as it continues to develop.

SA: Thank you guys for the partnership.

HT: Thank you. Thank you for the partnership. We’re really enjoying it.

GB: We are too.

Rightfully, the financial press and analysts have rung alarm bells of bubble speculation: circular financing, Oracle's 500% debt-to-equity ratio, and Meta’s off-balance sheet special purpose vehicle (SPV).

Even one of the ever-optimistic management consultancies (Bain) has questioned how the estimated $2 trillion needed for AI data centers by 2030 will be funded.

Yet, I predict that dealmaking is not over.

The compute jigsaw puzzle is still missing two critical pieces.

Where are the power and energy deals?

These AI chips will need power. Many gigawatts (GWs) of power. Without it, GPUs are just $50,000 paperweights in a server rack.

Where are the new chip fabs?

As far as we know, TSMC is the primary chip fab service provider capable of mass-manufacturing bleeding-edge AI chips at an efficient volume. Fabs take years to build, and their future production capacity has long been reserved by the likes of Apple, NVIDIA, and Google.

Economically useful inference: Structural form of production in the AI Era

As I have publicly written about before, AI inference (a technical term for an AI model’s process of producing output) is the structural form of production in the AI era.

More importantly, we should care very deeply about attempting to measure the amount of useful intelligent inference over time. This will be the best indicator of AI’s long-term trajectory. However, “useful intelligence” is subjective to the user and the surrounding context.

Meanwhile, tokens processed can be a noisy proxy for understanding the progress toward more intelligence. Without a doubt, this metric is continuing to grow explosively—mostly constrained by the ability of AI providers to match demand.

In May 2025, Google disclosed its AI workload grew ~50x year-over-year, processing around 500 trillion tokens per month across its products.

From my previous estimates:

Today’s global tokens processed is doubling every ~3 months⁴!

To continue to serve global demand this year, we’d need to add 25 Colossus data centers in a year’s time.

Each xAI Colossus consumes ~300MW of power.

25 Colossus-sized data centers require about ~7.5GW of power, or roughly enough to power ~3x San Francisco-sized cities.

OpenAI’s dealmaking is directly responding to the expected growth in inference demand.

Modern Alchemy: Energy into Intelligence

In Vaclav Smil’s book “How the World Really Works,” he provides a powerful analysis of how human progress is tied to energy consumption.

“An average inhabitant of the Earth nowadays has at their disposal nearly 70 times more useful energy than their ancestors had at the beginning of the 19th century. [...]

Translating the last rate into more readily imaginable equivalents, it is as if every average Earthling has every year at their personal disposal about 800 kilograms (1.8 tons, or nearly six barrels) of crude oil, or about 1.5 tons of good bituminous coal.

And when put in terms of physical labor, it is as if 60 adults would be working non-stop, day and night, for each average person; and for the inhabitants of affluent countries this equivalent of steadily laboring adults would be, depending on the specific country, mostly between 200 and 240.”

In today’s world economy, energy consumption is a critical factor in the goods economy—but less so in the service economy. AI will soon flip this paradigm.

Two quick illustrations:

1. China, the world’s manufacturing center (~30% of global manufacturing), has grown its electricity generation capacity at a compound annual growth rate of ~8.5% since 2000. In 2024 alone, it added more renewable energy capacity (in 1 year) than the entire installed renewable power capacity of the United States.

2. TSMC, the world’s primary supplier of cutting-edge AI chips (~90% market share), is projected to consume ~12% of Taiwan’s total power generation by the end of 2025. TSMC contributes about ~8% of Taiwan’s GDP.

Today, most AI infrastructure projects face supply constraints: chips, land, data center shells, networking parts, etc…

But the constraint that overwhelms all others is power.

Power dictates winners in AI

The U.S. has had roughly stagnant power generation capacity for the last two decades. Like much of America’s aging infrastructure, power transmission capacity upgrades have been behind schedule for roughly the same amount of time.

Reading between the lines of the recent NVIDIA/OpenAI and AMD/OpenAI deals, GWs of data centers require GWs of new power generation capacity.

This is the most revealing tell:

NVIDIA: To support the partnership, NVIDIA intends to invest up to $100 billion in OpenAI progressively as each gigawatt is deployed.

AMD: The first tranche vests with the initial 1 gigawatt deployment, with additional tranches vesting as purchases scale up to 6 gigawatts.

The mutually agreed-upon benchmark for progress is to bring GWs of AI data centers online. Therefore, if we track inference as a measure of AI progress, then tracking power usage will be a major determinant of success.

Over all time horizons—the short term (today), the medium term (2-4 years), and the long term (5-10 years)—the story will be the same1.

Power unlocks compute. Compute unlocks AI. AI unlocks intelligence.

Smil’s insights still stand in the era of AI.

How does the world really work? Energy.

Foundry details missing: Samsung/SK & NVIDIA/Intel

OpenAI is planning to add ~20GW of data centers over the next decade. Sam Altman blogged about adding ~1GW every week as a longer term goal.

Today, the cost of AI chips makes up ~50-80% of the total cost of ownership of a new AI data center.

With OpenAI opening a dual-sourcing approach to its supply of AI chips (making deals with both NVIDIA and AMD), the question remains of how TSMC will be able to absorb all of the demand from the chipmakers.

Most advanced AI chip manufacturing is done by TSMC. TSMC produces bleeding edge chips for NVIDIA, AMD, Apple, and Google, the supply is already earmarked.

New fabs take around 19-38 months (~19 months in Taiwan, ~38 months in the U.S.) to be built. Supply will take time to come online.

Samsung is the closest competitor with fabs in Korea and the U.S. (soon to come online) capable of manufacturing bleeding edge chips.

So the final puzzle piece:

• Which of the remaining foundries (TSMC, Samsung, or Intel) will attempt to pick up the gap in supply? How quickly can they ramp up production?

• Who is going to guarantee their demand if the price of chips fall through the roof (due to increase in market supply)?

“We have no moat”, said an infamously leaked Google memo. The memo’s author was opining on Google’s place in the raging AI race. 

I think the memo is right in the sense of AI model development.

For sustainable competitive advantages among AI firms, Google most definitely has a moat: a two-decade head start on the cost of production learning curve (we’ll get to this at the end).

Large language models (LLM’s) are the foundation of modern artificial intelligence, AI. When ChatGPT launched in late 2022, the unintended smash hit was powered by a LLM called GPT-3.5.

Today, these models are trained on the world’s written and recorded information: text, photos, videos, and audio. 

Predicting the speed of technology advancement and adoption is notoriously hard. One of the major idiosyncratic risks in venture capital investments is “time”. When will AGI arrive? Who knows?

Small club of frontier AI labs

The universe of frontier (bleeding edge) AI model makers is very small. 

In the U.S., you can count these AI labs on one hand: OpenAI, Anthropic, Google DeepMind, XAI, and Meta. 

If you included non-U.S. labs and U.S. stealth labs1, the universe perhaps increases by 5-10 firms depending on how you count. 

Everyone is building the ~same thing!

Set the scene for the world’s most expensive natural experiment in competitive dynamics. 

For every frontier lab, at the most foundational level, the goal is the same.

Build the next frontier (most intelligent) AI model and hopefully reach AGI (artificial general intelligence). 

In addition to the “end” product—the AI model—being very similar, the resources “consumed” by the labs are also very similar. 

The labs basically need 3 types of resources: (1) research talent, (2) GPUs, and (3) data.

The scarcest resource is talent. The news of $100M offers from Meta and acquihire deals—$2.4B Google/Windsurf deal and $14.3B Meta/Scale AI deal—cement the intense competition and transferability of talent.

Next, GPUs are critical. To train and run AI, the labs need 100,000’s to 1,000,000’s of NVIDIA’s latest GPUs2. To use these GPUs efficiently, purpose-built data centers with sufficient power and connectivity are being built at breakneck speed.

• Elon Musk’s XAI built a $10B+ data center in Memphis in ~130 days called Colossus.

• OpenAI is partnering with Oracle to build multiple $10B+ data centers across the U.S. Their Wisconsin data center will be the largest single building datacenter in the world.

Finally, the last piece is data. Earlier models use very similar datasets, from the public internet. More recently, much of the model performance gains derive from data creation, evaluation, and generation techniques.

These data techniques are closely guarded trade secrets, as these recipes can easily enable another lab to take the lead. Meta’s recent Llama 4 model stumble can be partly attributed to inferior data quality and techniques. 

Captive resources: the race to keep scaling these resources at all costs

The playbook to make frontier models is simple:

1. Hire many cracked AI researchers

2. Give them 100,000’s of GPUs that are wired to run large training jobs

3. Give them the best data in the world to feed the models

Every frontier lab is pushing the pedal to the metal on all 3 levers. 

In that case, what are the sustaining competitive advantages for these model makers? Will everyone arrive at the same outcome? Are models simply all going to converge? 

Ultimately, AI businesses are no different, delivering intelligence at massive scale will become a firm’s sustainable competitive advantage. 

But what does it mean to deliver intelligence at scale?

Economically useful inference supplied

Intelligence is hard to measure. It is relative to the subject task.

For now, the proxy metric can be measuring total tokens processed and generated (aka inference supplied). To be more specific, it is the quantity of economically useful inference consumed by the market. Sorry, building the world’s largest bean counter with GPUs will not count.

As the basic form of AI “production”, the amount of inference equates to the quantity supplied in classic microeconomics.

To understand why inference is so important, let’s breakdown how these LLM’s work.

If you already are familiar, you can skip to the “AI in the wild: Inference at scale” section.

The makings of an LLM: training models

Pre-training, mid-training, and post-training are the colloquial stages of training. 

At each stage, the machine learning objectives and algorithmic optimization techniques are slightly different.

Mid-training and post-training stages are some of the most closely held secrets of modern frontier AI labs. 

A glimpse into pre-training: the neural networks that make up these models are tasked with predicting the next “word” in a set of “words”. 

Next token prediction, as it is called, is the primary objective in pre-training. 

The quick brown fox jumps over the __??__

As the model is trained to predict the next token in every sentence available on the internet, the neural network becomes a highly compressed version of the internet. 

What does it mean to “train” a model?

During training, each attempt at predicting the next token is called taking a “step”. 

In every “step” of training an AI model, three key components come into play: 

1. data sampling (picking a sequence of words from a dataset, and hiding some words)

2. inference (predicting the hidden words)

3. optimization (how close are the predicted words to the actual hidden words? update the model to be better next time)

We won’t deep dive into the optimization step in this post, there are many great posts on how it works. Backpropagation is the “magic” in machine learning optimization.

Seemingly very simple, researchers at frontier AI labs like OpenAI, Anthropic, Google DeepMind, and Meta toil over every minute possibility to improve these three parts of training. 

The goal is to train an increasingly powerful and intelligent model, at ever greater compute efficiency.

Inference is AI: it is predictions all the way down

When a user prompts a chatbot like ChatGPT or Claude and hit Enter, the model takes the prompt—the input data—and conducts inference to predict the most likely word.

Basically, this is the first two steps of the training process: 

1. Prompt: input data

2. Inference: generate predictions

Of course, these chatbots are performing more than predicting the next word. 

What happens under the hood is the process is continuous (or called auto-regressive). After the first word is predicted, it becomes the new “input data” for the next prediction.

The quick brown fox jumps over __??__

The quick brown fox jumps over the __??__

The quick brown fox jumps over the lazy __??__

The end result is a coherent and intelligent response from the AI.

AI in the wild: Inference at scale

Inference—the actual usage of AI—is the structural form of production from AI, not training. 

In May ‘25, Google disclosed that their entire product surface was processing ~500 trillion tokens per month. In April ‘25, Microsoft was processing ~50 trillion tokens per month. OpenAI and Anthropic both have not publicly disclosed their inference supply. According to Google, their tokens processed metric grew ~50x year-over-year. 

But if we were to infer from the Google and Microsoft figures, and make some assumptions:

• OpenAI’s supply is ~3x Google’s scale.

• Anthropic is ~1x Google’s scale.

• XAI is ~1x Google’s scale.

• Meta’s internal product inference is ~1x Google scale.

• All other inference providers (including Microsoft) is ~1x Google scale.

• Including Google, total inference supply would be ~8x Google scale.

Today’s inference supplied is probably somewhere around ~5-10x Google scale, equating to around 2,000 to 4,000 trillion tokens processed per month. 

Let us assume a more moderate growth rate of 25x YoY (vs. Google’s most recent 50x YoY), then that leads us to some crazy implications.

Imagine if today’s global inference demand requires ~250k H100s3 (very low estimate intentionally). To give a sense of the scale of ~250,000 H100 GPUs, xAI’s Grok 4 model was trained in Memphis on a cluster of 200,000 Hopper GPU’s called Colossus. 

Today’s global tokens processed is doubling every ~3 months4!

To continue to serve global demand this year, we’d need to add 25 Colossus data centers in a year’s time.

A different kind of learning curve for scale

In machine learning, researchers are used to monitoring learning curves. The curves help researchers understand if the model is learning. 

As training continues (more training steps with data/tokens), the model should become more accurate in its prediction—meaning the loss (error) metric should go down. 

In this curve for the Kimi K2 model, you can see the model gets a lower loss as more tokens are processed in training.

In industrial and manufacturing settings, a different kind of learning curve is used to analyze the unit cost of production as cumulative output increases. 

This is an actual learning curve from a HBS case describing one of DuPont’s chemical plant’s unit costs (in labor terms). To reach costs of ~7 work hours per metric ton produced, DuPont had to cumulatively produce ~0.6M metric tons of chemicals over 10 years. 

For any new entrant to the market, the only path to achieving similar unit economic cost is to catch up in cumulative production. 

In the case of attempting to compete with DuPont, they would have to somehow catch up ~10 years of production.

Google does have a moat: their cost advantages are emerging

When Google’s internal memo leaked saying there is no moat, that is partially correct. 

In terms of producing the best AI model, the inputs are commoditized and transferrable (as discussed above as talent, GPUs, and data).

Any new entrant can acquire those inputs (likely at increasing costs as supply dwindles) to competitively compete in model development. 

Yet, Google’s moat is not in model development. 

Its moat is from serving inference: deploying AI models at scale to the world. This has always been Google’s core mission: using machine learning to organize information. 

While Google Cloud Platform (GCP) is a laggard in the hyperscaler cloud business, Google (like Meta) has decades of experience running data centers for machine learning workloads.

Google’s ML workloads are mostly internal, and their data centers are purpose built to be the best infrastructure in the world.

XAI’s Colossus cluster may compete in compute capacity, it is not even close in operational efficiency when compared to Google’s data centers.

Using the industrial learning curve analogy, Google has cumulatively produced over 2 decades of ML workloads. This form of production is most similar to today’s LLM workloads. 

The leading provider (OpenAI) has a lot of catching up to do.

While some of Google’s past ML learnings are not directly transferrable, most of their distributed computing learning are likely already being applied at scale for inference.

It is no wonder that their models are at the Pareto frontier of cost-performance for all leading AI models.

Overcapacity: massive stumbling possibility

As OpenAI, Anthropic, and XAI races to build GPU capacity for training and inference, they have a massive stumbling possibility to their economics.

Sudden overcapacity of GPUs is possible. If one of the frontier labs creates a AI model that step-change from the competition, demand can very quickly dissipate from the previous leading providers.

While demand will follow the best supply, GPU’s sitting in data centers cannot.

Tight rope for the new AI labs to walk, managing enough GPUs for research, for inference, while not oversupplying in case of disruption.

No signs of slowing down

While ChatGPT user trends may be skewed towards the most tech-forward population, firms across America are adopting AI in an accelerating pace.

Roughly two years ago (a few months after ChatGPT was released), only ~4% of U.S. firms were using AI. Today, almost 1 in 10 firms (~9%) used AI in the last 2 weeks, with ~12% planning to use it in the next 6 months.

For the information sector— the internet, communications, and media firms, the adoption rate is ~27%. One in four information firms used AI in the last 2 weeks.

Generally, white collar services sector are adopting AI at a faster rate than the national average (~9%). Professional services have an adoption rate of ~22%.

Across the United States, 12 states are leading the charge with >10% statewide adoption. Colorado, Arizona, and Washington are the top 3 states with adoption rates >12%.

At Google’s recent I/O conference, Sundar revealed that Google’s AI workloads have increased ~50x over the past year. This includes all Google’s product surfaces: i.e., Search, Gmail, Google Docs, etc…

In search, Google revealed their AI overviews had more than 1.5 billion users, much more than ChatGPT.

All the while, AI models continue to get smarter with more compute power

In OpenAI’s latest release of its reasoning models, the company continued to demonstrate algorithmic progress that scaled AI models’ performance with the amount of computation used during training.

AIME is annual invitation-only high school mathematics competition. At this point, AI models are surpassing expert human performance on these exams.

Pound for pound, models are getting cheaper for the same smarts

While models are getting smarter with more compute, with each subsequent generation, they are also getting cheaper to use.

Not only are labs pushing their models’ intelligence frontiers, they are doing so with greater cost efficiency.

Sectors leading AI adoption already displacing labor

Especially in the technology and software sectors, executives are already providing guidance that their workforce will be smaller because of AI.

Amazon’s CEO Andy Jassy recently said in a company memo:

Today, we have over 1,000 Generative AI services and applications in progress or built, but at our scale, that’s a small fraction of what we will ultimately build. We’re going to lean in further in the coming months. We’re going to make it much easier to build agents, and then build (or partner) on several new agents across all of our business units and G&A areas.

As we roll out more Generative AI and agents, it should change the way our work is done. We will need fewer people doing some of the jobs that are being done today, and more people doing other types of jobs. It’s hard to know exactly where this nets out over time, but in the next few years, we expect that this will reduce our total corporate workforce as we get efficiency gains from using AI extensively across the company.

Similarly, Shopify’s CEO Tobias Lütke said:

Before asking for more Headcount and resources, teams must demonstrate why they cannot get what they want done using AI. What would this area look like if autonomous AI agents were already part of the team? This question can lead to really fun discussions and projects.

The change dilemma for non-tech organizations

It is easier for large software firms to adjust their enterprise processes on a dime, as most processes are build around malleable software that can be easily integrated with new AI systems. Teams that adopt AI systems faster are also those with higher competency in adapting software to optimize processes.

For large organizations operating in sectors where technology is not necessarily a core competency, managers and employees may have opposing incentives.

With the breakneck pace of AI developments, even the most tech forward companies struggle to keep up. Non-tech organizations have to trade off managing existing operations and trying to develop a new competency in technology. The trade-off is rarely worth the risk for the average manager.

For example, in clinical health care organizations, firms are short-staffed and overworked on the frontlines. Managers and employees are struggling to meet patient demand with existing resources. Their primary incentives are to deploy resources to incrementally improve care capacity and outcomes. Usually, the levers are to hire or reallocate staffing, purchase more machines, invest in more clinical space, etc.

Even if a hospital manager wants to explore AI in an existing setting, it will require updating procedures and re-training staff. All of which will impact care capacity in the short term.

For employees, some may view their own jobs as at-risk for AI displacement. For these folks, they have no incentive to support AI initiatives and projects. As the saying goes, no one wants to train their own replacement.

As a result, the pace of AI diffusion across sectors will likely be uneven for the years to come.

New firms with less mature processes and organizations will gain higher leverage from new AI tools. There are less barriers to experimentation, and processes can be changed overnight.

Mature firms must break through the organizational inertia to adopt change with a risk-adjusted mindset, or risk falling behind. For now, managers at firms like Amazon and Shopify are attacking this inertia head on from the top down.

It is a country with unprecedented market competition. Companies compete on price and cost intensely, and often race to the bottom until sustainable economic profits are non-existent.

What is competition? What does economics tell us about competitive dynamics in a perfect market?

If we take a step back to Economics 101, microeconomics theory teaches us a very simple dynamic: 

If markets are perfectly behaved, as more suppliers enter to match demand:

Supply = Demand, and economic Profits = 0.

Let’s sell some hot dogs to understand why this is the case?

Imagine this scenario of hot dog stands in a baseball stadium. 

1. Initially, you are the only hot dog stand (Only Hot Dog Stand) in the stadium, you charge $3 (price) for the hot dog. 

2. Hot dogs sell like hot cakes, baseball fans love hotdogs. You sell 1,000 hotdogs each game, and it only costs you $0.50 (cost) per hot dog.

3. In a single game, you make $2,500 in profits (profits = 1,000 x ($3 - $0.50)). It’s an amazing business, there are almost 80 home games per baseball season. 

4. That’s $200,000 in profits a year, for 1 baseball season that only runs ~50% of the year!

But after 1 season, an eagle-eyed shrewd season ticket holder Anne (with some capital at her disposal) realized your hot dog stand is very profitable.

Anne decides she’ll also open a hot dog stand (Anne’s Hot Dog Stand)! Anne’s Hot Dog Stand also charges $3 per hot dog. 

The stadium’s fans are usually the same people, and a hot dog is a hot dog—there is no difference to fans. Fans end up splitting their purchases 50% / 50% between your Only Hot Dog Stand and Anne’s Hot Dog Stand. 

Suddenly, your profits at Only Hot Dog Stand just tanked by 50%, you’ll only make $100,000 each year, instead of $200,000.

That is terrible news! So you ask around with the stadium’s fans. What would it take to get their business back?

The fans are motivated by their wallets, if Only Hot Dog Stand charges $2.90 instead of $3 per hot dog. All of their business will come back!

Okay, perfect, Only Hot Dog Stand lowers price to $2.9 per hot dog and recovers 100% of the stadium’s business. You lose about $8k/year of profits because of the lower price, instead of $100k/year. That seems like a great trade!

But terrible news, Anne also gets wind of this price change, and matches your price! Now, you’re back to splitting profits 50% / 50% at a lower price.

Long story short, if this cycle continues, Anne and your hot dog stand will race to set prices lower and lower, until you both make $0 profits. 

You’ll price hot dogs as close to $0.50 until you both decide the stand is no longer worth the trouble. Now, no one gets a hot dog.

Case in point: China’s domestic EV market 

In essence, this exact hot dog stand dynamic is happening across nearly all industrial markets in China. 

Electronics, construction materials, apparel, plastics, car manufacturing, and logistics. In almost every single manufacturing and industrial sector in China, there is intense competition.

The competition is most apparent to consumers and foreigners in China’s domestic EV market. There are ~140 EV brands in China’s domestic market, and only ~20 are believed to be heading towards profitability by the end of 2030.

As of 2025, BYD is the world’s largest EV manufacturer, beating the innovative leaders like Tesla to surpass USD $100B annual revenue.

Recently, BYD decreased the price of their cheapest pure-EV model Seagull to ~$7,780 USD per vehicle. A new 4-door fully electric hatchback (with ~190 miles of range) for less than $8,000!

In April 2025, BYD sold about ~55,000 Seagulls in its domestic market in 1 month. By comparison, in the US and Canada, Tesla sold about ~45,000 vehicles in 1 month in the most recent quarter.

These price decreases triggered a new price war, similar to the hot dog stand price war. Impacting their profits, China’s EV stocks collectively fell amid news of more price discounts.

How do China’s massive car manufacturers pay for some of these price cuts? Unsurprisingly, they use the same tactics as massive US retailers (Walmart, Costco, etc..), they squeeze their suppliers.

In 2023, BYD and NIO (another Chinese EV company) waited almost 260 days before paying their suppliers for delivered parts. 

Imagine, if you had to wait 8 months before receiving your first paycheck after starting a new job!

China's central government was equally wary of this race to the bottom dynamic. Recently, it summoned the country’s top EV leaders, “urging” them to decrease payment terms. Soon after, Chinese EV makers individually publicly announced a return to 60-day payment terms.

In the case of China, there are no exits

In our hot dog stand case, the Only Hot Dog Stand and Anne’s Hot Dog Stand raced to a bottom price of $0.50 per hot dog. When the price war approaches the lowest price, it is very unlikely both hot dog stands will remain in business.

Anne and her shrewd business sense will likely decide to close up shop and not sell hot dogs for $0 profits.

However, in China, bankruptcies and business closings are much less frequent. Larger manufacturers, that were initially capitalized by loans from local city and provincial governments, very rarely close up shop. 

Even when the company fails to generate profits, the plants continue to produce in order to keep workers employed. These zombie car plants are noticeably apparent, even to China’s central government.

Dayun, a China truck maker in inland China, is one of these zombie car companies. Its upstart in Shanxi (an inland rural part of China) uplifted the ancient city out of poverty and opportunity. Local governments bought ~33% of Dayun’s trucks in the 3 years before its IPO in 2020. 

Without Dayun, Shanxi’s growth targets would be much more difficult to achieve. Its workers will go unemployed, and local politicians will be between a rock and a hard place (the people or the party). 

Deflationary times ahead for China

China’s intense competitive dynamics is entirely deflationary. Competition drives prices down, and consumers experience lower prices.

Deflation can be just as harmful as inflation. Just ask Japan. The Bank of Japan tried for many, many years to stimulate spending with a negative interest rate. But people would rather lose money in the bank account than to spend it. They hope prices will keep decreasing, and it would be cheaper to spend in the future!

China’s cheap labor force provided an enormous deflationary force to the global economy since its entry into the WTO. Its current mature industrial capacity and prowess is always in search of profits.

When profits are swallowed up by new players, there are no more profits to be made. The cycle repeats.

Entering its 5th year of crisis, China’s property market was the first to fail in 2021. Its demise is in suspended animation due to government intervention. Opaque debt arrangements between China’s central bank, regional banks, local governments, and land developers prevent a total collapse of the market.

Book suggestion: Apple in China

If you’re looking for some summer reading, read Patrick McGee’s fantastic book on Apple in China. The electronics manufacturing prowess of China’s industries did not emerge from thin air. 

Corporate investments by giants like U.S,’s Apple and Taiwan’s Hon Hai Precision (aka Foxconn) transformed the entire global electronics supply chain, starting with China’s manufacturing labor force and expertise.

See ya next week, enjoy the Summer Solstice on Friday.

Good morning, bonds have been in the news recently. So here’s a more topical essay on how bonds work, and how foreign investors may utilize the financial instrument.

Fixed income assets (i.e. bonds) provide a crucial tool in global capital markets.

In its name, bonds’ fixed duration income streams provide a reliable method to match long term liabilities, i.e. life insurance and pensions. But doing so in a volatile foreign currency without hedging creates substantial risk for the bond holder when exchanging back to the home currency.

Traditionally, the US dollar provided a safe haven from large movements in currency exchange, but it is slowly becoming less true for Asian currencies that have seen a very volatile couple of months.

As a reminder, here is how a bond works.

A bond usually has 2 components: (1) a promise to pay the principal amount when the bond matures/comes due, and (2) a promise to pay a coupon (interest) amount regularly. Because both of these components are “fixed” (not changing) after the bond is issued, these are “fixed” income assets.

If you hold the bond to maturity, you will receive the principal and coupon payments, less any risk of the bond issuer defaulting. It is in essence a form of debt between the issuer and the holder. Sovereign nations and corporations are the most common issuers as bonds as a way to raise capital to fund operations or investments.

At the same time, most bonds are marketable. Before a bond matures, a bond holder can sell the paper to a willing buyer and receive a price for the remaining income and principal at maturity. The price is set by the market, which determines the current value of the bond’s remaining income stream, plus any premium demanded for any default risk.

• If country A issues a 30-year bond with 5% semi-annual coupon rate and principal of $1,000, the issuer (country A) is promising to:

  • Every 6 months, pay $25 in interest, for a total of $50 per year.

  • At the end of the 30 years, pay $1,000.

• The price of this bond issue is then determined by an auction.

  • In this example, if the price (determined by auction) is lower than $1,000, it means the bond has a yield greater than ~5%. The bond purchaser will earn the 5% coupon rate, plus the amount between the price and the principal.

  • If the price is determined to be greater than $1,000, the bond will have a yield less than ~5%, with a possibility to be negative. As the bond purchaser will pre-pay some amount for the future income stream.

• Bond prices move opposite of yield. When yield increases, bond prices fall.

Therefore, bond holders are often in one of 2 buckets: (1) held to maturity bond holders—i.e. the investor fully expects to hold the paper for all remaining income streams, and (2) available-for-sale asset holders—i.e. the investor expects some possibility they may sell the bond before it matures to another buyer.

Life insurers, pension funds, and banking: Some uses of bonds

The goal of life insurance and pension funds is to provide a promise for some “fixed” payout far in the future. This payout may be a lump sum or a recurring fixed payment.

Therefore, in essence, as an intentional over-simplification, pension funds and life insurers simply have to match their future liabilities (obligations) with the current funds and reserves.

Fixed income assets become a perfect tool for this job.

A bond’s future income streams are pre-determined/fixed, and the risk of default for high credit corporations and sovereign nations are low. Sovereign nations can always print money, and high credit corporations like Microsoft or Pfizer have fortress balance sheets with low leverage.

For example, for a pensioner that is soon to retire (with expected life expectancy of 30 years remaining), the fund simply has to buy a 30-year bond that provides sufficient coupon/interest payments to cover the recurring pension payments, then the liability is fully funded.

Beyond insurers and pension funds, some banks also use long dated bonds as a way to generate income (but some would say speculate).

For example, First Republic Bank failed in 2023, they very often used customer deposits to buy long dated US treasuries (i.e. 30-year bonds). Then, the bank claimed they will always hold the treasury to maturity (i.e. they have no intention to sell).

Unfortunately, the bank was forced to sell much of their treasury holdings when there was bank run on their deposits. They were insufficiently capitalized to handle the withdrawals. They did not have enough cash liquidity, because their capital was either locked up in loans to customer (mortgages) or bonds (treasuries).

Taiwanese/Japanese life insurers and sudden currency appreciation

Most recently, amid the trade war uncertainty and volatility, the US dollar weakened significantly against most global currencies.

In particular, Asian currencies appreciated much more quickly than normal.

This means, for foreign investors, like Taiwanese or Japanese life insurers, their US dollar investments suddenly decreased in value relative to their home currencies.

New Taiwan Dollar appreciates ~8% in 1 month

In the case of Taiwanese life insurers, firms recently had to take huge losses.

Over one month, the NT dollar surged more than 8%! They had to keep paying out life insurance policies in Taiwanese dollars, but their portion of US Treasury income suddenly dropped (when converted to NT dollars)!

This is hugely problematic for life insurers that invested their reserves into US treasuries. Taiwan’s life insurers hold 70% invested assets in foreign currency.

After taking into account the currency exchange, the fixed income stream from US treasuries no longer matched their liabilities! The difference between their obligated life insurance payments and funded claims became a loss.

Imagine you had 100,000 yen of funded payouts, and 25% was originally fully funded from US treasuries. If the US dollar declined 5% against the yen, then the insurer incurred a 1.25% (5% x 25%) loss, or 1,250 yen.

While the example amounts seem small, insurers usually do not carry too much excess reserves. These types of fluctuations can bankrupt them, because the underlying insurance liabilities are very large. Goldman Sachs estimates that a 10% appreciation in New Taiwan dollars will deplete the insurers’ foreign reserves and create solvency concerns.

The currency appreciation cycle is also self-reinforcing. When a Taiwanese life insurer sell US dollar assets to cover NT dollar liabilities, they (1) sell US treasuries, and (2) buy NT dollars with US dollars. Their US treasuries selling action drives down prices further. When they buy NT dollars, they drive up NT dollar appreciation further. Partially, this is what caused the massive and quick spike in NT dollar appreciation as they covered losses.

Yields for long dated Japanese bonds skyrocket

Japanese life insurers faced a similar issue and are facing billions of dollars in losses, though their exposure to US assets is marginally lower than the Taiwanese insurers.

However, the Japanese have another problem, even when most of their reserves are funded against their home country currency, their own government bonds’ yields are having a roller coaster moment. Their 30-year bond yield rose ~1% in less than almost a month.

A ~3% yield for long dated Japanese bonds had not been seen since the 1990’s!

Remember, for bonds, prices move opposite of yields. When yields rise, bond prices fall.

Normally insurers that are buying bonds for held-to-maturity, yield volatility for long dated maturities would not impact them. The underlying fixed income stream does not change assuming the default risk does not change. The underlying insured liabilities are still funded.

But think, how realistic is it for the insurers to really hold these bonds to maturity when there is substantial currency volatility?

Insurers risk another First Republic Bank moment during times of currency volatility. Their demand for liquidity can lead to bond sales. Insurers are often held to a lower capital reserve standard than banks.

If the Japanese yen continues to experience unprecedented appreciation against the dollar, these movements and the resulting paper losses will force insurers to sell held-to-maturity bonds! They have to reduce leverage to remain sufficiently capitalized.

Unfortunately, if that cycle takes place, the same self-reinforcing phenomenon would create a contagion effect across the global capital markets. Japan is the largest foreign holder of US treasuries.

If they must unwind their foreign and local bond holdings to reduce leverage and maintain capital ratios, the selling activity will drive US treasury and Japanese government bond yields higher.

When yields rise, prices fall. The resulting contagion effect is hard to stop.

In the Japanese bond market, the Japanese central bank already holds ~50% of all Japanese long dated treasuries; this was the result of almost 2 decades of quantitative easing amid Japanese yen deflation and low growth. Bank of Japan is quite aware of their limited tools for market intervention.

In the US market, US treasuries are already seeing signs of tested demand levels. With rising US deficits and interest payments, the US must continue to issue treasuries to fund the government. The final backstop would be a Federal Reserve action to purchase treasuries in the open market to support prices.

When central banks purchase bonds on the open market, they do so with their asset side of the balance sheet. On the asset side of the central bank, they simply “create” money into the system.

In other words, the only backstop is for central banks to print money.

Large bathtub, big spillover: where will foreign reserves go?

There has been more and more talk of foreign investors looking for non-US assets to diversify away from the current US risks.

But foreign investors hoping for alternatives to US assets have limited options.

In 2023, MSCI estimated the global investable market to be ~$271 trillion dollars. Investable assets generally includes securities and investments that are easily accessible by institutional investors.

The US market is about ~44.5% of the global investable market:

• Fixed income: ~$44.7 trillion

• Public equities: ~$44.6 trillion

• Private equities ~$2.3 trillion

Think about the global capital markets split up into different bathtubs of different sizes.

Each country’s economy gets a bathtub in the size of their investable market.

But with these bathtubs, you cannot change their sizes overnight. They change sizes very, very slowly.

It requires economic development and demographic changes to change the size of the investable economy. For some economies, the demographic trends are on an absolute downtrend (i.e. South Korea and Japanese demographic trends). For others, the majority of the economy is an export economy (i.e. Taiwan). These factors take decades to change.

Investors that try to bail water away from the US bathtub to other bathtubs (including their own) will find only one outcome.

Other bathtubs will overfill and spillover. It will destroy real value if not done in a gradual and careful manner: more risk taking without adequate opportunity for returns.

For now, foreign investors are stuck and the taps will keep flowing. But the big elephant in the room remains, the next crisis will likely be a crisis of currencies. It will be a return of currency volatility of the 1970’s to 2000’s. When a stampede starts, it will be difficult to stop it.

See ya next week.

At the beginning of May, I wrote about a few structural issues that the U.S. will face as it navigates past “Liberation Day”:

1. The permanent damage to the American brand

2. Dual-sided (supply and demand) supply chain shocks caused by tariff uncertainty  

3. Acceleration of shift in global security and geopolitical center of gravity

4. Fiscal discipline and accelerating U.S. government debt burden

Here is a quick update on the American brand, while we’ll slowly cover developments in the other points later in June.

The invaluable American brand

A concept that may be obvious to most financially minded folks: the value of a company’s brand is often the company’s largest asset. 

Yet, this “intangible” asset value is often not formally included in modern financial accounting and reporting. 

You will not find any “dollar” value on these companies' balance sheets for the brand value of Apple, Tesla, Coke Cola, and other iconic American brands.

“I think I’ve done enough”: Brand value destruction at Tesla

The most obvious example of self-inflicted brand value destruction is Tesla. As the leader of DOGE, Elon Musk did not do Tesla any favors by appearing in the White House’s cabinet room and oval office.

In Tesla’s Q1 ‘25 earnings, Tesla reported car deliveries fell ~13%, and also missed Wall St consensus deliveries by ~53k cars.

It was Tesla’s worst quarter since 2022, giving back almost 4 years of progress. 

Earlier this week, Mr. Musk has finally come to his own conclusion that, “I think I’ve done enough.” He was referring to his political contributions and activities. 

Well, unfortunately, what’s done is done. There is unlikely to be an easy do-over.

In a few quarter’s time, EV market share figures in the U.S. will show how much permanent damage has been done. Anecdotally, I think Rivian trucks and SUVs have taken over San Francisco’s streets.

China has overtaken the U.S. in favorability around the world

China is now more popular than the U.S.

According to two independent studies, the 2025 Democratic Perception Index and the Morning Consult’s daily tracker both show a broad change in popular sentiment away from the United States.

The shift in sentiment is broad and global. As you can see in the map below, China has grown more popular than the U.S. in many democratic and affluent European nations (Scandinavia, France, Germany, Spain, etc..). 

Not only, the shift is also apparent in America’s closest neighbors: Canada and Mexico.

The permanence of the White House’s trade actions have started to take shape. 

At last week’s G7 meeting of finance ministers and central bankers in Canada, European Central Bank President Christine Lagarde said: 

"International trade will never be the same again."

“What sounded like a complete reshuffling of the cards as of April 2nd [of] this year has also launched a complete review of relationships, of trust, of diversification of sources and destination of both products and services”

Canada hosts the G7 meetings this year. Mark Carney, Canada’s newly elected leader, also gave a likewise sobering speech:

“Our old relationship with the United States, a relationship based on steadily increasing integration, is over. The system of open global trade anchored by the United States – a system that Canada has relied on since the second world war, a system that while not perfect has helped deliver prosperity for a country for decades – is over.”

Well, that sucks.

Harvard University: Attack on America’s thought and academic leadership

In light of Trump’s attempt to cut off international students from Harvard University, the Editorial Board of the Wall Street Journal wrote a rebuke of Trump’s attack on Harvard.

“This will be terribly damaging to America’s ability to attract talented young people who bring their enterprise and intellectual capital to the U.S. Non-citizens accounted for more than half of doctoral degrees in AI-related fields in 2022. Many have gone to work at U.S. companies like Nvidia or started their own.

The National Foundation for American Policy finds that “immigrants have founded or cofounded nearly two-thirds (65% or 28 of 43) of the top AI companies in the United States, and 70% of full-time graduate students in fields related to artificial intelligence are international students.” Immigrants have also started more than half of America’s privately-held startups valued at $1 billion or more.

Even if it’s modified, Ms. Noem’s order will echo around the world as a signal that the U.S. is no longer open to educate the world’s brightest young people. Foreign students will get the message and take their talents elsewhere. China’s politburo must be laughing at their good luck that their main adversary is hamstringing itself—first with tariffs that make its firms less competitive, and now with an assault on immigrant talent.”

China already spends more government funding on research than the E.U. and the U.S.

At a moment in time when the U.S. should be doubling down on science and research, the U.S. is deciding to cede another position to China. 

China is not wasting any time. 

Peking University and Tsinghua University (both top universities in China) are already closing deals with top scientists from Europe and the U.S. 

New AI and science labs with guaranteed funding will attract unnerved scientists attempting to clear the grey “funding” cloud on their lifelong career.

The TACO theory?

The growing complacency in the U.S. equity markets (as markets rebounded very quickly from its April lows) suggests there is a growing Wall Street consensus on the TACO theory: Trump Always Chickens Out.

If you believe most of this will be backpedaled and trade dynamics will only be marginally higher than January 2025, then you probably believe a recession is unlikely this year.

But I do not buy this theory. There is an extraordinary flight to quality assets.

Costco’s current EV/EBIT forward multiple is ~40x, while Walmart has a ~27x multiple in line with the likes of Nvidia and Microsoft.

This tells me that the Street has very little confidence in earnings more than a few quarters ahead. But because of Costco and Walmart’s fortress non-cyclical business in retail, they are happy to give them less of a discount in future value. Their retail positions give them tremendous pricing power to outperform rivals in stagflation scenarios.

Add to that, last Monday, Jaime Dimon gave his annual Q&A at JPMorgan’s investor day. and he said:

I am not a buyer of credit today. I think credit today is a bad risk. I think that people who haven't been through major downturns are missing the point about what can happen in credit.

When the world’s largest lender is leaning towards less lending, I say that is a bad omen.

See ya next week.

While the U.S. and China recently paused most tariffs on May 12, the OOCL Violet arrived in Long Beach a few weeks earlier on April 25. With ~$564 million of Chinese goods on board, Bloomberg estimates the goods on board will be slapped with ~$417 million in tariffs. 

A picture is worth a thousand words. One container ship, ~$1 billion dollars of trade.

Wild.

Designed by Apple in California, Assembled in China

Those famous words are sketched on the back of iPhones and MacBooks sold around the world. 

To explain the impact of imports on American businesses and consumers, it is helpful to go through an illustrative example of how spending on imported goods breaks down. 

When you buy an iPhone (an imported product), where does your money go?

An iPhone 16 Pro Max retails for $1,199 in the U.S. 

TD Cowen, an investment bank, estimates the phone has a ~$485 bill of materials (BOM) cost. Manufacturing and assembly costs about ~$57, which is mostly labor costs. The camera, screen, memory, frame, and other components cost about ~$438. 

The BOM cost is the total price that Apple pays for the iPhone to be manufactured, and assembled, and made ready for shipping. This cost excludes any duties (tariffs), logistics (shipping), and other costs after the product is assembled. It also excludes any indirect costs of the phone, i.e. the costs for Apple to design and market the phone. 

However, Apple does not get to keep the entire ~60% gross margins on the hardware. 

To bring a new iPhone to the market, Apple had to:

• design (R&D costs)

• market (SG&A costs)

• ship (logistics costs) the phone to the consumer

• Apple’s profits are also taxed by various governments (which includes U.S. federal and state income tax). 

Ultimately, if an U.S. consumer spends ~$1,200 on a new iPhone, ~$600 likely stays in the U.S. as opposed to going overseas.

~56% of spending for “Made in China” goods stay in the US

In 2019, the Federal Reserve Bank of San Francisco conducted a study on U.S. imports. 

For every $100 spent on imports, $42.6 stays in the US. 

For Chinese imports, the amount staying in the U.S. is higher. $55.9 dollars spent on $100 of Chinese imports remain in the US.

Buy the iPhone, please. When Apple profits, America profits.

The “Designed in California” portion of the engraving is certainly a critical aspect of the iPhone. 

When you buy an iPhone, most of your money goes to U.S. businesses. So if you want to buy American, buy the iPhone.

The American brand: What happens when foreigners don’t visit?

In a previous issue, I mentioned how foreign tourism has dramatically declined since Trump’s attack on global trade.

Recently, the World Travel & Tourism Council (WTTC), a trade association, estimated the U.S. will lose ~$12.5 billion in direct foreign tourism spending in the US. 

Not only that, Bloomberg reports, “Out of 184 global economies analyzed by WTTC in conjunction with Oxford Economics, it’s the only one projected to lose tourism dollars this year.” 

The projection represents a ~6% year-over-year decline in direct tourism spending in the US. 

A ~$50 billion trade: From deficit in “goods” to deficit in “goodwill”

The Bureau of Economic Analysis (the U.S. agency responsible for measuring the U.S. GDP) estimates that tourism represents about ~3% of the U.S. GDP. In 2024, nominal GDP for the U.S. was $29,184 billion.

Doing some rough math, the ~6% annual decline in tourism translates to a decline in GDP of about ~$53 billion. Keep in mind, the U.S.’s trade deficit with China is only ~$294 billion in 2024.

The U.S. may simply be trading its deficit in “goods” for a deficit in “goodwill”. 

You can decide what is a good trade.

Expecting higher inflation, consumer sentiment sinks to near record low

Wow, U.S. consumers really are not optimistic about the future.

Inflation expectations continue to skyrocket, hitting levels unseen since the 1980’s.

Remember inflation rates and inflation expectations are very closely related, to the extent that it is almost a self-fulfilling prophecy.

Expecting higher inflation, workers bargain for higher wages. Sellers list their products and services with higher prices as a result, and consumers spend more today expecting items to cost more tomorrow. 

Early signs of worsening hard data on consumer credit

At the same time, in addition to soft data (as surveys are known), hard data is still mixed on the state of the economy.

At least for the portion of consumers that rely on revolving credit card debt (usually the consumer segments that are most disadvantaged), delinquencies continue to climb. 

Credit card delinquencies do not reflect well on the state of overall consumer health, but often provide early signs of trouble ahead. 

Walmart, world’s largest retailer, is increasing prices

In its latest earnings call on May 15, Walmart CFO said they have already started to pass on cost increases to consumers.

Despite the pause on U.S.-China tariffs, Walmart prices are expected to continue to increase through May and June. That is because a third of Walmart’s goods are still imported from abroad, and import tariffs are still ~13-15% on average.

From Bernstein and the Economist:

Before the recent trade truce with China, which has resulted in American tariffs being lowered to 30% for 90 days, Bernstein estimated that new import duties were on course to raise the cost of products sold at Walmart by an average of 5.2%. Suppliers, it reckoned, would absorb half of that, with the average sticker price in Walmart stores rising by 2.6% to make up the balance. Sales volumes would drop by 2% in response, giving revenue a net boost of around half a percent. 

Thanks for reading. See ya next week.

As a programming note, I decided to dedicate more time to writing weekly updates that will hopefully be published on Monday’s. If you have suggestions and feedback, please email me!

Geneva trade talks: they are talking! 

“Candid, in-depth, constructive” - Chinese Vice Premier He Lifeng

“Substantial progress” - U.S. Treasury Secretary Scott Bessent

After a weekend of official face-to-face talks, both American and Chinese representatives signal a non-adversarial tone to the discussions.

I think these talks kick off the first innings to a series of discussions, very similar to the talks that took place from 2018 to 2019.

Remember, in the 2018-2019 trade spat, in-person bilateral trade talks started in earnest around May 2018 in Beijing. The “Phase 1” trade deal wasn’t finalized until around December 2018, after almost seven months of negotiations.

While both sides’ language from this weekend’s talks sound optimistic, the global context has substantially shifted. It is also round two in this bruising boxing match, and one side has had much more time to prepare: China.

The 2020 “Phase 1” U.S.-China trade deal was mostly a marginal reset of 2018-2019 tariffs, with both sides agreeing to pause tit-for-tat tariffs and progressing to mutual semi-annual discussions in “Phase 2”. 

Phase 2 of the trade deal never took place. The Biden administration lost hope of any more progress after the Chinese ultimately did not meet their target of purchasing $200 billion of additional U.S. goods–mostly farm goods and commodities. 

The White House claims a trade deal has been struck at this May 10-11 set of talks. Yet, the Chinese’s state media has only promised a joint communiqué1 on May 12, alluding to an agreement on the mechanisms for restarting trade consultations based on “mutual respect”. 

Meanwhile, over the same weekend, Chinese President Xi flew to visit Russian President Putin in an official state visit, presiding over the May 8 Victory in Europe (VE) day. Moreover, this week, South American leaders from Brazil, Colombia and Chile will visit Beijing to meet the Chinese leader to re-energize relations between the region and China. 

Talks are better than no talks. Actions speak louder than words, so time will tell whether it is real talk, or just a bad rerun of the 2018 trade talks.

Commonwealth buddies: Australia and Canada

Last week, another Commonwealth country rejected right-wing politics. Similar to the recent Canadian victory of Canadian’s Mark Carney, Australians voted for the governing Labor party and gave the incumbent Anthony Albanese another term as prime minister. 

The resurgence of Australia's Labor party in the polls had a similar trajectory as Carney's reversal of fortunes–voters in Australia and Canada rejected Trumpism on the global stage. Peter Dutton, the leader of Australia’s opposition party, similarly lost his own seat–exactly like Canada’s Conservative party leader Poilievre.

Mark Carney also wrapped up his short visit to the White House, with little more than:  

“it's not for sale, it won't be for sale, ever.”

Americans can have their Range Rovers again

Back in the British Isles, from a Jaguar Land Rover factory in England, British Prime Minister Starmer announced the U.S.-U.K. deal. 

Remember, the U.K. is one of the few nations with whom the U.S. has a trade surplus.

But nonetheless, what is in the U.K. trade deal you ask?  

Practically, there is nothing of significance.

Tariffs are most the same–with the sole exception of cars. Americans can have their Land Rovers and Range Rovers again after the export pause. 

US small businesses are getting more vocal after the pain

In my last update, I touched on how small businesses will bear the brunt of the trade war: ~80% of U.S. employment is by small businesses. 

Since the higher interest rate environment started in 2024, the health of small businesses has been declining.

For the last 13 months, total employment for businesses with less than 10 employees declined every single month.

Businesses have also started to fall behind in their credit card payments. 

The contrast in tone between big and small business is stark. 

With Blackstone and Goldman Sachs taking over entire floors at the Waldorf Astoria in Beverly Hills, one financier at the Milken conference last week told the Financial Times,

“This is the most self-censored Milken conference I have ever been to.”  

Yet, the brave small business owners are speaking out.

From the Wall Street Journal:

“Nobody in power seems to care about small business,” said Scott Anderson, owner of 5 Star North, which works with Chinese manufacturers to make its products ranging from acrylic markers to tiki torches. “At this point the only option I see is selling out the rest of what we have and shutting our doors.”

From the Financial Times:

The owner of a California-based food manufacturer, who asked not to be named, said one Chinese supplier “offered to change the cogs on invoices to help me evade tariffs” soon after Trump rolled out the increased duties.

“My option is to lay off my team or join in the fraud,” the owner said.

From the New York Times:

Tim Fulton and Ramper Innovations, a manufacturer of airplane equipment based in Sitka, Alaska.

He had an order from the U.S. Air Force that he was confident would serve as a catalyst and bring in new customers from Asia and the Middle East while luring potential investors.

“I feel like things have ground to a halt,” said Mr. Fulton. 

With no orders to fill, Mr. Fulton rented out his home in Alaska and temporarily moved to Brazil, where his wife is from, because the cost of living is lower. And instead of closing deals with investors to raise more money, he is consulting a bankruptcy lawyer.

Yes, Made in America. Let’s stay open for business first?

Treasury Secretary Bessent told a room full of investors at the Milken conference: 

Hire your workers here. Build your factories here. Make your products here.

I agree. The U.S. must rebuild its capacity for domestic production and reinvest in its workforce. Build and innovate in America.

But as Dave Chappelle has said, “we want to wear Nike’s, I don’t want to make them.”

When the tariffs are putting an Alaskan airplane equipment manufacturer for the U.S. Air Force out of business, that is just plain bad execution.

See you next week.

Warren Buffett represents everything that is good about American capitalism and America itself—investing in the growth of our nation and its businesses with integrity, optimism and common sense. – Jamie Dimon

At Berkshire’s 60th annual shareholder’s meeting in Omaha, Nebraska, Warren Buffett held his last question and answer session as the CEO of Berkshire Hathaway. In the last 5 minutes of the shorter-than-usual 4 hour session, Buffett sprang a surprise to the world.

Greg Abel will become the CEO of Berkshire Hathaway by year’s end.

Berkshire shareholders will know, this is hardly a surprise. In 2021, Charlie Munger let slip that Abel will be the successor-in-waiting.

“Greg will keep the culture,” Munger said in response to a question about how to manage Berkshire’s growing complexity as a decentralized conglomerate.

GOAT

Many have written about Buffett’s legacy, and Munger’s.

The success of Warren Buffett’s $1.1 trillion empire is such that its scale overwhelms the S&P 500 return in a line chart, you can barely see the line.

New coach, new season

With ~$340 billion1 in cash built up, Berkshire has a war chest ready for this new era.

Target, Nike’s, Lowe’s, CVS, and United Airlines.

Berkshire’s war chest can afford to buy all 5 of these companies at today’s prices.

It is safe to say this chapter of American capitalism has come to a close, and there will never be another one like it. In finance, size is returns’ worst enemy.

Here are 4 hours of his final Q&A session as CEO

To the Oracle of Omaha, let’s raise a Coke and have a McDonald’s burger in his honor.

Liberation Day unleashed the next stage of trade warfare

Shortly after inauguration, the new US administration embarked on a new trade war footing. The US placed new tariffs on goods from China, Canada, and Mexico. Throughout February and March, the counterparties likewise have increased their tariffs in response in a tit-for-tat motion–with some occasional pausing and backsliding.

However on April 2nd (Liberation Day), the US announced broad tariffs on nearly every US trading partner, with an indiscriminate minimum tariff rate of 10% and higher. In more tit-for-tat motions, the US raised tariffs on China to a cumulative tariff of 145%, with China retaliating with a cumulative tariff of 125%.  

The new “reciprocal” tariffs exceeded Wall Street’s worst expectations. US equity and fixed income markets reacted dramatically to the broad protectionist measures.  

In about a week, the S&P 500 lost ~11%. US junk bonds yield increased ~100bps. The US 10-year treasury rate increased ~50bps, the largest weekly increase since 2001. The 10-year is frequently used as the world’s risk-free rate for capital markets, as a global barometer of investment risk appetite and trust in the US financial system.

While Trump suspended the reciprocal tariffs (10%+ broad tariffs) for 90 days for all trading partners on April 9, the US-China tariffs remained. As of May 1, 2025, the US and China representatives are not in any active bilateral negotiations.

2025 starts with economic contraction

In the first contraction since the first quarter of 2022, the US economy contracted 0.3% in the first quarter of 2025, mostly driven by imports front-running tariffs. The 2022 contraction had the backdrop of the highest inflation rates since the 1980s. In June 2022, annualized inflation reached a high of 9.1% . Around the same time, the Federal Reserve started tightening financial conditions to combat inflation: raising interest rates from 0% to a high of 5.5% in 2023. 

This time around, there are more signs of structural obstacles to the growth of the US economy, relative to the ex-US world.

1. The “US” brand as a symbol of free trade and safe haven has forever been damaged. “Liberation Day” and the fallout from the US trade policy progression continues to puzzle US and foreign investors. The lack of coherent US policy objectives creates tremendous uncertainty in global trade and development. 

   1. Year to date, the dollar has continued to weaken, with the US dollar index down ~7%. Tourism to the US sharply declined in the days after tariffs.

      

   2. Ex-US equities are up ~8.4% for the year, outperforming US equities by ~13 points.

      

   3. The bond selloff in the treasury and corporate bond market was mostly foreign driven, while the recent rebound in prices was mostly inflow from US domestic investors.

   4. The lack of policy coherence creates a large disincentive for 1-5 year capital investments in the US, especially as it relates to reshoring manufacturing and infrastructure capacity.

   5. For example, Apple, the symbol for US consumer brands, has no plans to move manufacturing to the US, but instead will shift more assembly volume to India, mostly as a means to side-step the US tariffs.

2. The supply chain shock from a man-made shock to the global trade system has yet to arrive.

   1. As much as the US administration may want to decouple global trade, it cannot be done overnight. 

   2. Similar to the COVID dual-sided (supply + demand) shock, the supply chain is undergoing a similar shock. Sea-bound containers from China to US have fallen 45% year-over-year in April 2025. 

      

   3. While US merchants front-loaded inventory, they are simply delaying the inevitable: a combination of (1) a shortage of goods and/or (2) a massive drop in demand due to increased prices due to tariffs. 

   4. This type of supply shock will impact small businesses the most, those who do not have the working capital capacity to soften the supply chain shock with increased inventory. Small businesses employ 80% of America and are responsible for ~40% of US imports from China. When small businesses suffer, America suffers.

   5. Even assuming the US and China come to some type of equilibrium that is able to restart trade–current conditions are tantamount to a trade embargo for most goods, the infamous bullwhip effect will come back into play. 

   6. The US inventory to sales ratio will give us a good indicator of how severe the shocks will be this time around, with March 2025 figures releasing May 15.

3. Shift in global security and perspectives is already becoming evident. Geopolitically, the US absconding from the global alliances and partnerships will have longer term consequences beyond one US presidential term.

   1. Canada: Most obviously, Trump’s policies have literally reversed the Canadian election. Enabling Mark Carney’s Liberal party to erase the Conservative party’s ~10-15 point lead over a span of 40 days. The Conservative party leader even lost his own seat.

      

   2. European security: Germany is gearing up for arms. Its defense industry base is being re-ignited after Germany approved a new 500 billion euro infrastructure fund. Shepherded by Merz–who was elected in Feb 2025 after Trump started the trade war and embarrassed Ukraine at the White House, the new law exempts defense and security spending from Germany’s strict debt guidelines–enabling the state to issue more Bunds to finance its security objectives.  German defense companies (i.e. Rheinmetall) have more than doubled in value since then. 

      

   3. Indo-Pacific security: Australia and Philippines will hold general elections in May 2025, on the 3rd and 12th respectively. Both countries play a crucial role in Indo-Pacific security, as a US allies and regional balance of power against China. The Australian elections have already been an intense balance between its Chinese and US ties in recent weeks. 

4. Pending new tax and budget legislation, US debt ceiling and fiscal outlook remain similarly uncertain as trade policy.

   1. The White House places a July 4th deadline to finish passing Trump’s tax agenda.

   2. The X date (when the US treasury will reach the debt ceiling) is likely to be sometime this summer or early Fall.

   3. With uncertain economic outcomes over the short and medium term, drastic changes in lowering tax receipts relative to debt burden will place additional stress on Treasury yields. 

The crossroads ahead

The most important major crossroad will on the path to US-China trade negotiations in earnest.

While the two parties have yet to talk, there are increasing unilateral signals that both sides are willing to sit down.

On May 2, China (via its state owned media CCTV) has signaled some willingness to approach U.S. under the right circumstances. But China’s other arms of state continue to message a strong stance of no concessions with the US.

The backdrop for these prospective talks are not good: (1) uncertain economic growth, (2) declining consumer confidence, (3) prospects of skyrocketing prices, and (4) uncertain US fiscal and tax outlook.

Events to watch

• May 3: Australian Parliament elections

• May 7: Federal Reserve interest rate decision

• May 12: Philippines midterm elections

• May 15: U.S. March 2025 inventory-to-sales survey data

• May 28: Federal Reserve May meeting minutes release

  
  
  
  

Almost 9 months later, suddenly, public markets decided that NVIDIA should be worth ~$500B less. Okay…, there was a catalyst, DeepSeek released R1, their o1/3 comparable model the week before.

Not an overnight affair: Random walk with upward drift

Yet, the progress was gradual; an o1/3 class model in the public was an eventuality in 20251.

Modern ML is mostly based on intuition, gut, and grunt work–betting on big ideas and empirically validating the results. One idea, built on top of the last useful one, without a clear line of sight on the next research breakthrough.

Most AI researchers are still compute/GPU constrained (or some people like to say the GPU rich vs. the GPU poor; maybe not the best turn of phrase). There is often a mile-long laundry list of experiments they wish they can implement/run on any given day. But not enough compute.

With some luck and sheer grunt work, researchers often stumble on new clever ideas that work. Folks have learned a long time ago (The Bitter Lesson) that letting machines learn is often the best way. By sheer coincidence, another team of researchers (from the Hong Kong University of Science and Technology) converged on the same R1 reinforcement learning findings as DeepSeek, published only a few days apart.

DeepSeek’s journey training LLMs did not start in 2024, it started long before as a side project at High Flyer2 from around 20193.

From DeepSeek V1 to R1: Leveraging other open source research

DeepSeek’s progress emerged from a sea of impressive and accelerating research. Mistrial and META’s open source stance only accelerated their progress. You can clearly see DeepSeek took many architectural and scaling guidance from the other labs (size of training runs in tokens trained, hyper parameters, etc..).

Timeline of highlighted milestones related to DeepSeek LLM development

• JUNE 27, 2023: High Flyer announces proprietary internal HAI-LLM training framework

• JULY 18, 2023: Llama 2 released - 2T token runs - 7B to 70B models

• AUGUST 9, 2023: Reports of Chinese cloud providers stockpiling GPUs

• SEPTEMBER 23, 2023: Mistrial 7B - rumored 8T tokens - beats Llama 2 7B

• OCTOBER 23, 2023: Biden administration phase 1 GPU restrictions effective

• DECEMBER 11, 2023: Mixtrial of Experts - open source MoE 8x7B (12.9B active)

  • Likely compounded/leveraged on top of their Mistrial 7B runs/checkpoints

• JANUARY 5, 2024: DeepSeek V1 - 2T token runs - 7B & 67B models

  • Very similar to Llama 2 runs, uses GQA but deeper instead of wider models.

• MARCH 8, 2024: Gemini 1.5 - close source acknowledgement of MoE in frontier model

• MAY 7, 2024: DeepSeek V2 - 8T run - 236B MoE (21B active) close to Llama 2 70B performance

  • DeepSeek starts varying from the pack substantially…

    1. New form of attention: Introduced MLA, improvement alone ~80%+ reduction in KV cache memory requirements (compared to comparable GQA). When combined with other memory optimizations, DeepSeek claims 93.3% reduction in KV cache.

    2. New form of MoE: Introduced more flexible form of MoE with shared experts while still using auxiliary loss for load balancing

    3. With these improvements, compared to a dense 67B model, DeepSeek claims ~578% inference throughput improvement.

• JULY 31, 2024: Llama 3: META continuing their scaling program to 405B while increasing to 15T of tokens trained.

• DECEMBER 27, 2024: DeepSeek V3: scaling up 15T run 671B (with 37B active) close to frontier models, GPT-4o, LLama 3 405B, Claude 3.5 sonnet.

  • DeepSeek continues their drive for training and inference efficiencies, faced with chips constraints (primarily memory bandwidth, as H800’s have ½ the bandwidth of H100’s)... the path now is clearly their own:

    1. Improved MoE training: without auxiliary loss (learned routing, vs. heuristic)

    2. Multi-token prediction objective during training: taking a lead from speculative decoding, though not used in inference

    3. DualPipe training pipeling/scheduling framework: reduces bubble and communication bottlenecks during training

    4. FP8 training dynamics: while accumulating in full precision without substantial loss in quality to reduce memory bottlenecks

    5. NVIDIA SM’s allocation adjustments: Low level adjustment to allocate SM’s only for communication to reduce bandwidth bottleneck

    6. Modular inference infra4: 2 inference infra setups specialized in (A) prompt processing (input tokens) and then (B) sampling (output tokens) to separate and scale to fit workloads and optimize distributed batch processing at scale.

• JANUARY 22, 2025: DeekSeek R1: R1 model built on top of DeepSeek V3, close to OpenAI o1 performance and enables test-time compute regime

LLM development is a compounding phenomenon.

Labs leverages their last generation of models and builds on top of them. This is seen countless times again and again.

• Mixtrial 8x7B is a MoE of their 7B (rumored).

• Llama 3 models use Llama 2 models for filtering and data curation.

• Gemini 1.5 teams used Gemini 1.0 generation models for evaluations, data curation, and hyperparameter extrapolations. Gemini 1.5 Flash distilled from Gemini 1.5 Pro.

• DeepSeek V3 used DeepSeek V2.5 models for data generation in post-training.

• DeepSeek R1 is built on top of V3 base model.

DeepSeek V2 release in May 2024 was the beginning of DeepSeek charting its own path, rather than simply imitating others.

Its V3 release very much cemented their self-efficiency in continuing to progress LLM research. From DualPipe, multi-token prediction, FP8 training dynamics without little loss in quality, and finally to providing 2 full pages of hardware “suggestions” to “hardware designers” (read NVIDIA), these are not behaviors of imitators.

R1: Reinforcement learning and test-time compute will accelerate inference demands

DeepSeek R1 claims to be on par with OpenAI o1/3 in benchmarks. LLM expert users are already impressed and running these models on different infra and model configurations (DeepSeek released R1 distillations down to 1.5B size, which can run on most modern MacBook Pro’s with 16+GB memory).

As reinforcement learning will require some method of evaluation (to give feedback to the system whether something is correct or not), inference as a type of workload will only continue to increase.

For OpenAI, in the middle of 2024 (before the release of o1), inference costs are already dominating training costs (~$7B in compute costs, ~$4B for inference towards ChatGPT). This shifts from training to inference workloads will accelerate, to include inference workloads targeted for research for test-time compute regime models (evaluations, sample generation, etc..).

Hundred of Billions of CapEx in 2024-2026: Will only accelerate model scale ups and intelligence progress

"I will say that Deep Learning has a legendary ravenous appetite for compute, like no other algorithm that has ever been developed in AI. You may not always be utilizing it fully but I would never bet against compute as the upper bound for achievable intelligence in the long run. Not just for an individual final training run, but also for the entire innovation / experimentation engine that silently underlies all the algorithmic innovations."

– Andrej Karpathy

With open replications of R1 findings either in progress or complete, inference demands will only accelerate. Any model can become a reasoning model; where you can adjust a “knob” to increase test-time compute to get a more reliable answer.

Labs will be more comfortable scaling up when inference is cheaper with new forms of attention (MLA) and MoE’s. The test-time compute regime gives consumers of models the power of accuracy as a function of cost.

How much do you care about a more accurate response for a given task? $1k? $6k? $15k? This can now be within your control.

"We're still in punchcard era of LLMs, designing prompts, copy pasting context around, hitting go, reading the thing, prompting occasionally. Pretty lame. If there are fewer than a few thousand tok/s of sustained throughput generated on my behalf do we even have AI"

- Andrej Karpathy

Making it easier to communicate -- Attention

If we take a short journey back in time, generative AI (LLMs, diffusion models, etc..) started to emerge in the late 2010's, primarily driven by the clever introduction of attention into existing neural network architectures.

At the time in 2014, researchers were still trying to make deep learning work for machine translation. Ultimately, attention was a clever mechanism to help models adopt a method of communication between parts of a sequence (whether its nodes within the same vector i.e. self-attention, or nodes across different vectors i.e. cross-attention, etc...). Since 2017 transformers formally appeared as a new neural network architecture, it has transformed into a powerful tool partially because it was possible to parallelize the computation of attention across the modern computing accelerators (GPUs, TPUs). Then, it was possible to make bigger models, and train them on more data... i.e. scaling pre-training.

Scaling large language models was a slow start and took almost half a decade: the GPT's (1-3), BERT, T5, LaMDA, PaLM, etc... from 2018 to 2022.

But progress has been tremendous, and the quality of models has improved dramatically from new entrants and incumbents alike: GPT-4, GPT-4o, Claude, Gemini, Llama, Mistral, Cohere, Qwen, DeepSeek, O1, etc... until today.

Measuring progress, the year of benchmarks and evals

In addition to the cost of inference, measuring progress is becoming more difficult as the models are getting better and better, and beating many contemporary benchmarks.

To measure a model's ability to reason with acquired knowledge from training, MMLU was released in 2020; at the end of 2024, as models are reaching 90+% on MMLU. In 2024, MMLU Pro was released to provide more difficult tasks to replace MMLU, and in early 2025, models were already scoring 80+% on MMLU Pro.

For coding and software engineering, SWE-bench was released as realistic real world software engineering tasks in 2024, and in early 2025, models were already scoring 70+% on SWE-bench.

Leading AI labs have always been transparent about engaging the community to create new novel benchmarks/evals to evaluate their models. Recently, some of these labs have been criticized for waiting to disclose funding to fund new benchmarks. Personally, I doubt there was any bad intent.

It's clear that existing benchmarks are being saturated. Progress is increasingly subjective to the type of task and domain. 2025 will be the year of evals; perhaps new standards and systematic methods will emerge between the labs and the community.

Curse of success -- scaling inference supply to meet demand vs. research progress

Frontier AI labs have had tremendous success in distributing their models to the public. OpenAI expects $3.7B and $11.6B in sales in 2024 and 2025. Azure, AWS, GCP, Oracle Cloud, and others had fantastic AI-driven tailwinds in their cloud computing businesses1.

For research organizations like OpenAI, there is a rising tension between compute for inference vs. compute for research.

At the same time, the cost of measuring progress is becoming more difficult--and costly. At the end of 2024, for the o3 model's ARC benchmark, OpenAI spent at least $1M-2M2 to run a single benchmark (once!).

Through model development, it's understood as a training run progresses, evals are run at intervals to measure progress of training. If we assumed that while training o3, it took 5 ablation experiments, for each experiment it took the equivalent of 10,000 training steps. Then, if the ARC benchmark was run at every 1,000 step, then it would have run 5 x 10 = 50 times. 50 x $1.5M is $75M (!!). While this is pure speculation in the details, it's a good proxy to understand the cost of measuring progress. For $75M (which is some X% of the cost of o3 development cost, as it does not include GPU hours for training the model), one can train a GPT-2 class (circa 2019 generation LLM) model ~110,000 times3.

Yet, there's a lot of research progress to be excited about.

Progressing transformer architectures

Simply within the transformer architecture, DeepSeek's developments (mostly from DeepSeek (V3 and R1) in MLA (new form of attention that enables much smaller KV cache while maintaining performance of MHA), MoE routing without auxiliary loss (taking inspiration to allow models to learn routing vs. heuristics), multi token prediction (incorporating the insights from speculative decoding), RL pipeline for reasoning with R1 (progressing to the first open source test-time compute frontier class model) show that a small team that's not distracted with inference demands can accelerate the rate of experimentation from ideas4. Their DualPipe distributed training improvement shows GPU continued algorithmic improvements in resource allocation/scheduling to maximize hardware use (they also made other clever adaptations like FP8 mixed precision to reduce memory use, as H800 has lower memory resources).

Meta's findings with byte latent transformer shows that there may be a path away from fixed tokenizers and vocabs and they can be learned. Tokenizers can be blamed for a lot of problems with LLMs (i.e. the infamous how many R's are in the strawberry question). This could also provide a path to consume more data in training without using heuristics to pre-process data into sequences.

DeepSeek's models and Meta's Llama both acknowledge using the previous generation's model to generate data to progress to the next generation of models, signs of compounding progress. Previous work compounds and yet open sourcing allows the entire community to progress at a lower global capital cost which accrues value to more of the community.

Reducing compute burden for the same tasks and performance

Other than continuing to optimize and progress the transformer architecture, there are other ways to reduce the compute burden for the same tasks and performance.

There are many teams working on a variety of bets, some cool examples:

1. State space models (non-attention based architectures): Mamba, rwkv, etc.. to reduce the attention quadratic runtime complexity.

2. Extreme quantization: BitNet models are 1.6 bit models (with 4 bit activations) which are much cheaper to run and can run on CPU's as matrix multiplications are removed.

For the leading AI labs and hyperscalers, they are racing to build more data centers designed for AI workloads. In the short term, compute remain in a supply crunch (primarily due to chips shortage--see NVIDIA stock price). Data center construction has skyrocketed with the inflow of capital. The next physical constraint will be energy.

Energy and AI

Vaclav Smil published a book in 2022 called "How the World Really Works: The Science Behind How We Got Here and Where We're Going". The short answer: energy. Taking aside how Smil may feel about the progress/solutions of the energy transition, it is clear that our modern world--particularly the developed world--runs on energy.

"An average inhabitant of the Earth nowadays has at their disposal nearly 700 times more useful energy than their ancestors had at the beginning of the 19th century.

[...]

Translating the last rate into more readily imaginable equivalents, it is as if an average Earthling has every year at their personal disposal about 800 kilograms (0.8 tons, or nearly six barrels) of crude oil, or about 1.5 tons of good bituminous coal. And when put in terms of physical labor, it is as if 60 adults would be working non-stop, day and night, for each average person; and for the inhabitants of affluent countries this equivalent of steadily laboring adults would be, depending on the specific country, mostly between 200 and 240."

-- Vaclav Smil, How the World Really Works (emphasis mine)

The continued adoption and development of AI will require more energy than ever.

In 2025, Microsoft announced $80B of capex for AI data centers. In January 2025, The White House, OpenAI, Softbank, and Oracle announced a $500B investment in data centers and energy over the next 4 years. Amazon expected $75B in capex in 2024 mostly related to AWS, only to grow in 2025. The trend is the same for other players like GCP, Meta, etc...

AI data centers are joining the ranks of other items in the energy transition (i.e. electric vehicles, etc...). Forecasts of AI energy demands vary widely (from ~2x to 5x current data center energy demands by 2030). BloombergNEF's Michael Liebreich lays out a more nuanced perspective in this new generation of data center growth: we've seen this before but market dynamics, stakeholders governance, and energy consumption efficiency will all play a factor at moderating demand and supply.

Personally, I'm optimistic that physical constraints won't bottleneck the rate of progress derived from adding more compute. Capital is moving rapidly to balance the supply, and energy constraints are likely to be mitigated by more algorithm and hardware improvements (similar to the 2000's and 2010's as cloud computing adoption took off, when similar stakeholders cried out for more energy, but as a proportion of US energy demand, data center energy consumption grew relatively gradually due to more efficient hardware and software design).

AI and the Physical World -- How Humans Communicate With Machines

In 2025, in the developed world, most consumers spend their time on their phones and laptops. Let's look at how humans do a common task: shopping on Amazon's mobile app.

Tapping on a phone screen: Shopping on Amazon

Each user interaction follows a carefully orchestrated flow:

1. Physical Input → Mobile OS

   1. User taps or swipes generate touch events

   2. OS interprets and routes events to the application layer

2. App ↔ Server Communication

   1. App sends HTTP/TCP requests to backend servers (i.e. Amazon ecommerce backends with product listings, ads, etc...)

   2. Servers process requests (database lookups, payment validation)

   3. Communication is via the internet backbone (i.e. TCP/UDP connections) that route through a host of hardware and network software; not to mention the cryptography to ensure the data is secure.

3. Server ↔ Services

   1. Servers coordinate with other systems (inventory, payments, authentication, etc...)

   2. Data flows through multiple service layers to aggregate the information needed to respond to the user's request.

4. Response → User Interface

   1. Results return to device via the internet backbone

   2. App updates UI based on new state

   3. User perceives change and decides next action

Each loop is tightly scoped and optimized, typically requiring only 100,000 to 100,000,000 FLOPs5 per iteration of this loop . The system is engineered for speed and responsiveness through these small, discrete steps, which are then repeated in very rapid succession hundreds or thousands of times in a normal user session.

The Human Factor: Bearing the Cognitive Load

While the visible computation is relatively lightweight, humans shoulder most of the cognitive burden:

1. Persistent and Context-Aware Computation

   1. Users must constantly perceive and interpret their environment

   2. Humans translate intent into discrete interface actions

2. Interface Navigation

   1. Humans learn and adapt to predetermined UI patterns

   2. Users bridge gaps between their intent, environment, time, and available actions

The system's efficiency comes from delegating most adaptive intelligence to the human user, keeping machine computation minimal but requiring significant human cognitive work.

Yet, the efficiency didn't come for free. On the other side of the user consuming the interface, the interface was designed by other humans on top of a stack of software and hardware to enable the loop: chipset, operating system, network, application, etc...

The fixed cost is amortized over the many users that consume the interface over its lifetime.

LLM Inference: A Different Paradigm

Large language models with 70B parameters take a contrasting approach, requiring 20,000,000,000,000 to 200,000,000,000,000 FLOPs6 per inference - 200,000-2,000,000x times more than typical app interactions. However, they enable open-ended, natural language communication in a single pass.

With a fixed defined vocabulary space, the model can consume sequences of arbitrary length (up to a fixed length) and output a sequence of arbitrary length (up to a fixed length).

This is unlike programming, where there are syntax constraints that will limit the runtime/execution of the program with the underlying stack.

Can new AI systems balance the communication burden?

The evolution of human-computer interaction may point to a future where we're gradually shifting cognitive load from humans to machines. Traditional interfaces require humans to:

• Learn specific interaction patterns

• Maintain context and state

• Translate high-level goals into discrete steps

The integration of LLMs into system architectures could happen at multiple levels:

1. Application Layer

   1. LLMs could augment existing interfaces as an intelligent assistance layer

   2. Natural language could complement rather than replace traditional UI elements

   3. UI's can be fluid and generated on the fly depending on input from the user and the environment

2. Framework Layer

   1. Web and mobile frameworks could incorporate LLM-powered components

   2. Development tools could use LLMs to generate more adaptive interfaces easily

3. System Layer

   1. Operating systems could employ LLMs for more intelligent resource management

   2. System calls and memory allocation could become more context-aware (i.e. hardware aware, workload aware, etc...)

   3. Kernel operations could adapt to usage patterns and requirements instead of using heuristics

Remembering attention is a form of communication

Large language models are marvels of deep learning. Modern human language is excellent at compressing information. Attention mechanisms allowed machines to learn via communication between many many different nodes of a sequence in multidimensional spaces.

If we move the unit of analysis from language and sequences to humans and the physical world, where else could we leverage the added benefits of learned machine communication?

In the LLM paradigm: Learned data processing vs. pre-determined data processing

Since the explosion of AI, industry has been predicting the explosion of data generated/stored/consumed. At 2024 NeurIPS, Ilya Sutskever predicted that the era of pre-training is over, but data continues to be the fossil fuel of the AI era.

But if we extend the previous section's analysis of human-computer interaction, we can see that data today is mostly generated and consumed in small discrete steps/iterations. Media content is slightly more continuous (i.e. videos, audio) but the data are still discretely packaged, collected, and consumed at the user's direction.

Since 2024, the frontier models have also expanded in modalities (both input and output; image, audio, etc...) and inference speed (i.e. realtime voice/video API's from OpenAI and Google).

There are applications of transformers where the inputs and outputs are more continuous in nature, such as in the classic self-driving car example: Waymo Research's 2024 work on transformer driven trajectory prediction–taking realtime continuous perception and scene data inputs to predict future motion trajectories of objects in the environment.

Yet, the data is still discretized and processed into features and tokens to be made into a sequence for processing; researchers spent effort empirically to find the best way to process the data into a sequence to achieve the best performance.

To add new modalities and formats of data to models, models have to be re-trained or trained for longer with the new data in a different way. The vast majority of this is still determined by humans based on empirical heuristics.

Perhaps, this is the reason why I have a lot of excitement around ideas like Meta's byte latent transformer. All data can be encoded into a sequence of bytes, and the model can learn to process the data in a way that is optimal for the task. The caveat is that this will likely require larger compute budgets than labs are willing to experiment with (especially when the current methods are working well for their existing use cases and users).

Other interesting questions/possibilities in the LLM paradigm

Because LLMs are so good at taking arbitrary inputs and generating usable outputs, if computation budgets are not a concern, it's natural to ask how we can use LLMs to generate arbitrary outputs on the fly all the time.

Instead of having to hire an engineer to write web applications, we can use LLMs to generate the web application (this is also related to the idea that software agents will probably be the 1st proven class of agents from LLMs).

As opposed to the fixed cost / economies of scale dynamics of current software development, LLMs can perhaps reduce the accessibility of software development to the masses; but at the same time change the cost dynamics of fixed vs variable costs in software.

This is unlikely to be adopted for all use-cases, there are many use-cases where fixed costs / economies of scale are still king (i.e. most of all of the web applications we use today). But there are certainly a long tail of un-designed bespoke software applications that can be invented (but was never invented because it had a user of 1, or the user was already doing it manually and charging a very high rate for the service, etc...).

Bespoke software will become more accessible; it already is an enormous market (think how much system integrators like Accenture make per year; they are the definition of providing a labor force to develop customized software for their corporate clients).

Its growth will be driven by fundamental factors as the technology starts its journey down the learning curve, lowering the cost in developing and deploying models towards every general application. In the next 3-7 years, generative AI models will be as commonplace as gradient boosting and decision tree models for prediction and classification tasks. 

By way of a simple analogy about the retail industry:

Within an apparel retailer, there are 2 critical business functions: branding & marketing, supply chain & distribution. High performance supply chain operations and distribution is often buttressed by the accuracy of the firm’s demand and supply forecasts. Over-forecasting demand could easily wipe out the firm’s quarterly profits. Under-forecasting demand will greatly limit growth.

Prior to the arrival of modern sample and compute efficient ML techniques (i.e. XGBoost and random forest) for predicting time series data, leading global retailers (i.e. Nike, P&G, LLBean) crafted demand and supply forecasting models by hand. Even in the early 2000’s and 2010’s, these firms hired enormous teams of business analysts and managers to fine-tune and manage supply chain and distribution forecasts. 

Yet, by the late 2010’s, a novice computer programmer can easily train an highly accurate demand/supply forecast model (based on gradient descent) with a few lines of code and a small historical dataset. These models power the modern e-commerce world (i.e. Amazon, Walmart, etc..) predicting the demand and supply of their product lines to the minute and hour.

While the large language models have disrupted the world, the underlying mechanics and intuitions have not changed. The primary difference is that transformer-based LLM’s require substantially more (1) compute, and (2) data; many orders of magnitude more of both ingredients for the model to learn as generalized models.

The barriers to development will continue to decline.

Compute supply is currently constrained. Yet, chip supply is classically cyclical (over the past 40-60 years), and it will soon enter a cycle of oversupply. There will continue to be systems engineering constraint problems to be solved; but I am hopeful that the current rate of capital investment will be sufficient to unlock these barriers (i.e. the physical practicality of a $10B datacenter vs. a $100B datacenter in terms of connectivity, power, and management).

Data will be a more multi-faceted and complex problem; there are at least 3 factors: (1) specialized/fit-for-purpose data, (2) volume of world representational data, and (3) systems engineering to ease the distribution and ingestion of data in large-scale training. 

Specialized data (#1) and volume of world representational data (#2) are somewhat orthogonal to each other, but it is important to understand them together.

Specialized data (#1) is a pre-existing constraint of ML for current users and firms; nothing has fundamentally changed about whether a firm or user has high-quality proprietary data or not. If a firm has historical data relevant to its products, then it is absolutely in the best position to use that data to develop and deploy generative AI applications.

World representational data (#2) is a continuous research problem. It is known that current datasets are not exhaustive and comprehensive of our world; yet there is some fear of “running out of data”. Capital continues to flow to firms which conduct foundational research for data formats, modalities, and techniques to represent our entire world. 

Yet, with the sliver of world representational data (mostly from the internet), leading research labs have already shown the tremendous potential of large models by the current releases. While there is likely some declining returns to scale in a continuous fashion. I am hopeful that there will be some step-level discontinuity improvement in this area from modalities and format research. For example, neural networks are notoriously bad at representing 3-dimensions, various tricks are utilized to navigate this problem; yet humans only experience the world in 3-dimensions. Humans learn representations with much greater sample efficiency.

Scaling up data and compute infrastructure (#3) is a classic systems problem that is being solved by firms like Databricks and leading AI labs. This is unlikely to be a primary constraint to value creation.

The accumulation of value creation from generation AI will accrue to the users and the rest of the global economy.

As these barriers decline, training and/or deploying generative AI models and applications will become commonplace. We are likely far from reaching the bottom of the learning curve for this category of technology. As a result, leading model builders today must rely on their distribution-driven economies of scale to grow the market and retain share.

Fortunately, because we are still the top of the curve and in the early innings, there is still much to do to realize this vision. So much has yet to be built, and many opportunities lie ahead.

What is the current target range for the Federal Reserve’s Federal Funds rate?

Recent fun news

Save on baggage, rent clothes when visiting Japan, says Japan Airlines (JAL). Travelers visiting Japan via JAL are being offered a new service called “Any Wear, Anywhere” to ditch their heavy baggage and pack lighter. Through the service, you can rent six tops and three bottoms for 6,000 yen, or about $45, for two weeks. The airline believes this experiment can reduce luggage weight per flight, leading to less fuel use.

“Speak Now”: Taylor Swift continues to re-record her old albums, releasing the latest version on July 7, 2023. Ms. Swift has since re-recorded 3 out of the first 6 of her albums. Shamrock Holdings—a private equity firm owned by the estate of Roy E. Disney—bought the rights to those albums for $300M in 2020. Perhaps unsurprisingly, each release of a re-recorded album lifts Taylor’s entire catalog in the streaming charts (original and re-recorded). As music right holders are paid by streaming counts, each releases provide a large injection of music sales for both Ms. Taylor and Shamrock Holdings.

Jobs report: U.S. job creation in June was 209,000, falling short of the projected 225,000. This marks the first time in 14 months that actual numbers feel short of consensus, according to Bespoke Research.

Pension funds start unloading private equity assets. The New York State Teacher’s Retirement system is looking to unload $6 billion of assets into the secondary market. Private equity funds are usually close-ended for 8-10 years, these movements in secondaries suggest the smart money wants to take profits while the marks are still frothy.

This week’s Speed Read

1.    Some updates on office commercial real estate (CRE): Office commercial properties are started an upward trend towards high delinquencies since Jan 2023, from ~2% to ~4%.

2.    In Q1 2023, offices dominated CRE foreclosures (~63% of all CRE foreclosures).

3.    Higher quality office spaces fare substantially better, many metros (Boston, D.C., Manhattan) even continuing to see positive net absorption (incremental occupied squared footage) since 2021.

4.    CMBS (commercial mortgage-backed securities) market also signals similar discontinuities through bond spreads between b/w higher credit rating (Triple-A) vs. lower credit rating (Triple-B) bonds. The spread is ~6%, which is nearing March 2020 pandemic levels.

5.    Weekly bankruptcy filings moving back up to near GFC (2008 global financial crisis) and March 2020 (pandemic) levels.

6.    Inflation stubbornly not going away, with core PCE around ~4%.

7.    Wall Street continues to revise to higher inflation expectations in 2023 and 2024. 2023 consensus estimate at ~4.4%.

8.    Global temperatures hit another record high on July 6, 2023, 17.23 Celsius.

Speed Read Charts

This week’s trivia question answer

On May 4, 2023, the Federal Reserve’s FOMC (federal open markets committee) increased the federal funds rate by 25 basis points to 5.00% to 5.25%.

As of Jul 9, 2023, the market has priced in a 93% probability that the committee will raise the range another 25 basis points at the Jul 26, 2023 meeting.

Which country has laid down the most undersea cables (by length) since 1990?

US, China, France, or Japan?

Recent fun news

This week, the White House unveiled a new push for “Bidenomics” (press memo). In Philadelphia a few weeks ago, President Biden apparently said he does not know “what the hell it is.” From the memo, it mentions jobs, middle class, infrastructure, and it keeps saying it is working?

More money for chips and NVIDIA. Money continues to flow to generative AI startups making foundational models. MosaicML exited for $1.3B to Databricks. Inflection AI announced finishing a $1.3B round of funding, valuing at $4B.

Cumulative funding raised by foundational model startups:

1. OpenAI $11.3B

2. Inflection AI $1.525B

3. Anthropic $1.5B

4. Cohere $445M

5. Adept $415M

6. Runway $237M

7. Character.ai $150M

8. Stability AI $100M

Orcas have been intentionally ramming into ships on the ocean. The behavior has spread from the seas around the Iberian peninsula to the North Sea between Scotland and Norway, almost 2,000 miles away. Many theories, no answers.

Wall Street continues to downsize. Goldman Sachs cuts 125 Managing Directors globally. JPMorgan cuts 40 dealmakersin North America. The Swiss bank UBS plans to cut 30,000 jobs this year.

Move aside Silicon Valley Bank, FDIC data shows Bank of America’s ~$100B in paper losses from its bond investments made during the pandemic-driven deposit increase. By comparison, SVB had a ~$16B loss in its held-to-maturity portfolio. Bank of America’s bet on securities is leading it to trail its leading competitor (see number 7 in Speed Read), JPMorgan—who decided to sit on its pandemic cash instead.

This week’s Speed Read

1.     The monthly mortgage payment for a new average purchase loan size is close to $3,000, nearly 2x from ~$1,000 to ~$1,500 range from 2000 to 2019.

2.     Every single inflation forecast from the Federal Reserve since June 2021 has been wrong.

3.     In emerging countries (i.e. Brazil, India, Mexico), the cost of capital to build renewable energy infrastructure (i.e. a solar far) is ~2x higher than developed countries—EU’s ~4% cost of capital vs. India’s ~10% cost of capital.

4.     For the first time, in Q1 2023, China topped Japan to be world’s largest exported of automobiles.

5.     Morgan Stanley thinks the US will see its first negative payroll month around the end of this summer (August/September).

6.     Since the start of the Ukraine war, ~25% of Russia’s crude oil continued flow to Western countries in the EU, the US, and the UK, through the conduits of Turkey, India, UAE, and China

7.     In Q1 2023, Bank of America’s net interest rate spread (the difference between the interest rate it earns from its loans and assets, and the interest rate it pays to depositors) was 1.43% compared to JPMorgan’s 2.04% (42% higher).

Speed Read Charts

This week’s trivia question answer

France has laid the most undersea cables, closely followed by the United States.

Each of these countries have laid more than 500 thousand kilometers of cables, more than 1.3x the distance from Earth to the Moon.

What is the world’s largest endowment fund? Find out at the end!

Recent fun news

Taylor Swift tours are very good for hotels, termed the “TSwift Lift”. ~80% revenue bump in Nashville, and 10-20% bump in large metros like Chicago and NYC. I guess scalpers are not the only winners. San Francisco hotel operators can rejoice once Ms. Swift arrives at Levi Stadium later in July.

Masayoshi Son returns with his signature hyped slides at this year’s Softbank shareholder meeting. AI will solve everything? Find its full glory here.

This week’s Speed Read

1.     Commercial credit card rates topped 20.1%, highest since ~1970.

2.     New business applications remain elevated at ~175% pre-pandemic levels.

3.     U.S. home listings continue to sink to lowest level on record since ~2012.

4.     Uneven home price level effects across regions: SF Bay Area generally down ~12%, whereas NYC neutral ~0%, Miami up 8%.

5.     ~80% chance moderate El Nino, ~50% chance strong El Nino this winter.

6.     Global sea surface temperatures have been abnormally high (+1.1 degrees from 40-year average) in 2023 so far.

7.    Since Medicaid started unwinding COVID-19 renewal provisions, ~1.5M enrollees have been disenrolled (~300k in Florida, ~150k in Arizona).

Speed Read Charts

This week’s answer

Ensign Peak Advisors, Inc manages $124B in assets for the Church of Latter-day Saints, also known as the Mormon Church.