Host: Uncle Moon
Guest: Zhibin Xiao

Source: English subtitles. Dialogue is grouped into paragraphs, retaining the conversation and opening highlights.

Speaker labels are inferred from context and have not been checked against the audio throughout. A dagger (†) marks passages with uncertain attribution. Timestamps come from the subtitles; short exchanges within one subtitle may share a timestamp.

Contents

Highlights

Zhibin Xiao (Guest) (00:00:00)

NVIDIA has the best execution in the chip industry. Annual deliveries of lithography machines are another bottleneck. People think going from 7 to 5 or 3 nanometers makes chips faster. But the main benefit isn't speed. It's power consumption.

Uncle Moon (Host) (00:00:19)

I've heard of a Silicon Valley company hardwiring models into chips.

Zhibin Xiao (Guest) (00:00:23)

The next great chip company will be an AI-native chip design company.

Uncle Moon (Host) (00:00:29)

You know all this inside out. Your mind holds a history of chips and a map of every company. Incredible!

Zhibin Xiao (Guest) (00:00:37)

A failed tape-out can cost hundreds of millions of dollars, and a year or two of work.

From chip design to AI software optimization

Uncle Moon (Host) (00:00:48)

Meeting 100 fascinating people in Silicon Valley. I'm Uncle Moon. Today's guest is Zhibin Xiao, a leading chip architect with over 20 years of experience. He entered Zhejiang's Chu Kochen College in 1999, earned his PhD at UC Davis, and has worked in semiconductors ever since. Welcome, Zhibin.

Zhibin Xiao (Guest) (00:01:06)

Thanks for having me, Uncle Moon. Hello, everyone.

Uncle Moon (Host) (00:01:10)

Let's jump right in. Your career has taken some fascinating turns. You started by building accelerators into CPUs, then moved into dedicated AI chips and chips for sparse computation. Now you've founded a systems optimization company that doesn't make chips. That shift makes me wonder: is the frontier of AI performance moving from chips to software optimization? How did you make that transition?

Zhibin Xiao (Guest) (00:01:43)

That's a great question. The short answer is yes. The focus is shifting from designing individual chips to designing systems, including software for heterogeneous systems. One thing I've been fortunate about over the past 20-plus years is that I've stayed in my field. My work has always involved chip design. More specifically, hardware-software co-design and architecture. I started in electrical engineering at Zhejiang University. What drew me to computer architecture and chip design? I took courses in computer organization and architecture. A well-known professor, Yale Patt, who invented this approach to CPU branch prediction, came from UT Austin to teach at Zhejiang. I took his summer course. That's when I fell in love with computer architecture.

Zhibin Xiao (Guest) (00:02:50)

Looking back at everything I've done over the past 20 years, from my degrees to designing CPUs at Oracle, where I designed database acceleration chips, then Alibaba, working on Hanguang 800 and AI chips, then a startup making sparse AI chips, and now this company optimizing AI infrastructure, it's all been about the same thing: making computation more efficient. My earliest work was on digital signal processors, or DSPs. As an undergraduate and master's student at Zhejiang, I worked on single-core DSPs. That started around 2002 or 2003. Programs like China's 863 Program supported homegrown CPU and DSP development. I was lucky to join one of those projects as an undergraduate. I stayed at Zhejiang for my master's. It was rare for a student to work on a major DSP design from scratch. My master's project was based on MIPS, a traditional processor architecture. We extended the MIPS instruction set with multimedia instructions.

Zhibin Xiao (Guest) (00:04:21)

The DSP's job was fairly simple: MPEG encoding and decoding for VCDs, including audio and video codecs. That was my master's project. For my PhD at UC Davis, I moved from single-core to manycore DSPs. We used many cores for stream processing, or dataflow computing. We applied it to communications and video codecs, as well as big-data queries and sorting. That streaming structure is very similar to today's dataflow AI processors. The idea is to keep data flowing on the chip instead of sending it off-chip. Data moves among the DSP cores in a manycore processor. Each DSP has very small instruction and data memories. An interconnect links them together. It was quite an advanced architecture at the time. You could tune each DSP's frequency and voltage independently. That was my PhD research. At Oracle, I moved on to accelerating database operations inside CPUs. Oracle had bought Sun Microsystems around 2015. I joined Oracle in 2017. We were still working on SPARC processors then.

Zhibin Xiao (Guest) (00:06:09)

Some viewers may not be familiar with SPARC. SPARC, IBM's PowerPC, and x86 were major instruction sets for server CPUs. SPARC servers could run for years without being shut down. With Solaris, they were extremely stable. After acquiring Sun, Oracle wanted to innovate. How could we do that? Oracle's core business is databases. We wanted to put database operations directly into the SPARC CPU, as a coprocessor or an accelerator unit, sharing the CPU's memory bandwidth and making full use of it. That would support in-memory databases. We used TPC-H, which database people may know. It's a benchmark with 22 queries. We implemented those operators in hardware, then designed an instruction set and a runtime, and integrated them with Oracle Database. It essentially became an accelerator for Oracle's query optimizer. It could speed up certain operations. I spent about five or six years on that at Oracle. Then I worked on Alibaba's first generation of AI chips.

Zhibin Xiao (Guest) (00:07:38)

After that, at my previous company, we designed sparse AI chips. Now I'm building ZFlow.AI. Across those experiences, we've seen Moore's Law slow down. Improving a single chip's performance or reducing its power use is getting harder. The bottlenecks are now at the system and software levels. If what I'm working on is no longer the main bottleneck, or plenty of AI chip companies are already tackling that problem, I should tackle the next bottleneck. That's at the system level. That's why we founded ZFlow.AI. We use architectural simulation to guide AI chip design and optimize AI systems. We bring model compilation, runtime scheduling, and system simulation together in one platform. Customers can explore optimization strategies before they have the hardware. AI clusters are getting larger and more expensive. Many systems engineers don't have access to that hardware. Our simulator gives them a practical tool for system optimization. That directly improves how efficiently AI models run on the hardware.

Zhibin Xiao (Guest) (00:09:22)

It also answers an important question: which model runs most economically on which hardware? The hardware might exist already, or it might not exist yet. We bring both cases together through simulation and automated profiling. So, to come back to your question: yes. The AI chip is no longer the main bottleneck. It's the system and software.

Uncle Moon (Host) (00:09:50)

Thank you, Zhibin. Listening to you, I realize you've lived through all these waves: VCDs, multimedia, big data, databases, and now AI. My own PhD was in multimedia, and I used VCDs growing up. Over the past 20 years, you've been part of so many technologies that shaped our lives, approaching them through computer architecture and chip design.

Zhibin Xiao (Guest) (00:10:18)

Exactly.

Uncle Moon (Host) (00:10:18)

It's remarkable. It feels like a sweeping history of the industry, with you quietly working away through each big change. That's how it strikes me. Do viewers feel it too? I work in software, not hardware, but I can appreciate how much you've contributed to architecture and chips across all those fields. You also said your latest shift reflects Moore's Law reaching its limits. As it slows down, you're doing more on the software side. We'll come back to each stage of that journey in more detail.

Zhibin Xiao (Guest) (00:10:50)

My company focuses on system optimization. But every company designing AI chips needs to do this work first. So if we see an opportunity, I'm open to designing chips again. We might even incubate a chip company.

Taalas and when dedicated chips make sense

Uncle Moon (Host) (00:11:09)

I see. If simulation shows that certain models fit a particular architecture, you might build that architecture yourself.

Zhibin Xiao (Guest) (00:11:16)

Exactly. Model architectures are also starting to converge. Suppose a model already meets our needs for a specific task, like a robot vacuum, or monitoring vehicles in a parking lot. If the task stops changing much, we can build a dedicated chip to accelerate it.

Uncle Moon (Host) (00:11:46)

I've heard of a company hardwiring models directly into chips. Is that the idea?

Zhibin Xiao (Guest) (00:11:50)

That's a Canadian company called Taalas. Its founder also founded Tenstorrent, another AI inference chip company. Tenstorrent's CEO is Jim Keller, the legendary chip architect. Taalas is taking a very aggressive approach. Why are investors still backing it? I think it reflects a broader trend. For a stable, specific task, hardwiring a model into silicon can make economic sense. Their architecture still has some flexibility. It isn't completely fixed. They can change certain metal layers and run the wafer through fabrication again. That's cheaper than a new tape-out with a full set of masks. So it is programmable, in a sense. But you still have to send the wafer back to the foundry for a metal fix.

Uncle Moon (Host) (00:12:56)

An outsider sees the spectacle. An expert sees how it actually works. You're so good at explaining these chips and the companies behind them. I can tell this is going to be a fascinating interview.

From chip design to mass production

Uncle Moon (Host) (00:13:10)

You mentioned that you might make chips again someday. As someone who knows very little about how chips are made, I'd love to understand the process, from design all the way to mass production.

Zhibin Xiao (Guest) (00:13:23)

Sure. The whole chip development cycle is extremely long. When we designed CPUs, design alone took about two years. Then came tape-out. With the system and software, getting to deployment could take three to five years. Recently, NVIDIA has really accelerated that pace. They're releasing chips every year, sometimes several in a year. Designing a chip starts like building a house: what do you want to build? A codec chip? A communications chip for a particular Wi-Fi standard? Or something more general-purpose, like a CPU or GPU? That decision is driven by the market. Or you're working with a cloud provider, a hyperscaler with a specific use case. Some cloud companies, for example, develop codec chips. Video streaming is an interesting example. Different phones have different encoding and decoding capabilities. So uploaded videos arrive in all kinds of formats. The service transcodes them into a standard format for storage.

Zhibin Xiao (Guest) (00:14:47)

When it delivers them, phones differ in performance and resolution, so it has to convert the video into different formats again. That calls for a dedicated chip. You start with the need. Doing all of this on CPUs would be very expensive. Many CPUs don't even have the required codec capabilities. You'd need lots of CPU servers just for transcoding. So the requirements come first. Once they're clear, the architect comes in. The architect balances PPA: power, performance, and area, along with cost. How big should the chip be? What should it do? How does it compare with competitors? Competition in chips is brutal. The leader usually takes most of the market. Second place gets a little. Third and fourth get the scraps, if anything. You also need market research. Your chip must still be competitive in a year or two. That's the architect's job, and the result is a chip specification. Next comes the microarchitect.

Zhibin Xiao (Guest) (00:16:03)

The specification might define an instruction set or a high-level arrangement of functional blocks. Once the structure and functions are defined, someone must turn them into hardware. A function like an FFT for a codec can be implemented in countless ways. Each implementation involves different trade-offs. Given the architect's constraints, the microarchitect develops a concrete design. Many microarchitects also write RTL, expressing the design in a hardware description language. Then the verification engineers come in. Before AI, a chip design company might have two or three verification engineers for every design engineer. They handle functional, formal, timing, and physical verification. Before the chip exists, you also use emulation, often with FPGAs. Or you buy a large emulator, such as Cadence's Palladium. Verification is critical. A failed tape-out can cost hundreds of millions of dollars, not to mention a year or two. It's that important. After the RTL is implemented, you run synthesis, then physical implementation, place and route, clocking, and power design.

Zhibin Xiao (Guest) (00:17:51)

You co-design with the system vendor: how will power and packaging work? Once that's worked out, the design goes to the foundry, such as TSMC or SMIC. That stage takes roughly three to four months. What happens during that time? They make masks from your layout. Think of it as printing patterns on silicon, layer by layer. The complexity of a chip depends on the number of those patterning steps. Today, that can mean dozens of steps, approaching a hundred or more. Three or four months later, the wafers come back. A wafer test company checks them for defects. This isn't functional verification. The wafer is cut into individual dies. The foundry is responsible for yield: how many dies are free of defects. Those dies then go to a packaging company. The packaged chips are mounted on system boards for testing, including functional checks and tests under various laboratory conditions. Then you send samples to customers. After that comes mass production.

Zhibin Xiao (Guest) (00:19:23)

So the whole process, including design, takes a long time.

Uncle Moon (Host) (00:19:27)

Wow. That sounds incredibly long and complicated.

Zhibin Xiao (Guest) (00:19:27)

It is.

Uncle Moon (Host) (00:19:31)

How long does it take to reach mass production?

Zhibin Xiao (Guest) (00:19:33)

Traditionally, we'd say three to five years. For NVIDIA, chip development takes roughly two years. Some chip families now have a new generation every year. But the development teams work in parallel. A chip released today may have entered development two or three years ago. So NVIDIA's B300 and GB300 were probably designed two or three years ago.

Uncle Moon (Host) (00:20:05)

Wow. That's such a long lead time. A small mistake could cause major delays and problems.

Zhibin Xiao (Guest) (00:20:13)

Exactly. There's design, wafer yield, and the supply chain, then packaging, testing, cooling, and software that must be ready at the same time. There's a lot that can go wrong. That's why verification is so important in chip design. Here's a story about Jensen and NVIDIA. At one point, they were almost out of money. He spent the money they had left on an emulator to verify their GPU. That let them check it before tape-out. Back then, verification tools were still quite limited. It often took several tape-outs to get a chip right. With better verification and emulation tools, most chips can now work on the first tape-out. Some need a metal fix, but that's normal in the industry.

AI in chip development

Uncle Moon (Host) (00:21:20)

I'm seeing startups use AI agents for parts of chip design and verification. What's your view of those companies?

Zhibin Xiao (Guest) (00:21:30)

I know that space well. Friends of mine are building those companies. I won't name names, but it's definitely a trend. Most AI work in chip design today is focused on verification. Verification resembles programming, so it's a natural fit for coding agents. You have inputs, outputs, and simulators that can check whether the result is correct. That's why there's so much activity in AI-assisted verification. There's less AI doing the initial design itself. For physical design, such as place and route or floorplanning, EDA vendors like Synopsys and Cadence already use AI extensively. So AI in chip design is definitely where things are heading. But there's still a gap in architecture: what chip should you build, and how should you define it? There are very few tools for that. Human architects still play a major role. The second issue is that established chip companies struggle to adopt AI agents because they already have their own workflows.

Zhibin Xiao (Guest) (00:22:51)

Many engineers don't want AI to replace the work they do today. Doing that properly means rethinking the entire workflow around AI. I think the next great chip company will be an AI-native company. Architecture, design, verification, simulation, physical design: all built around AI. Most chip companies today use AI mainly as an assistant. To realize its full value, you need a company organized around AI from the start.

Uncle Moon (Host) (00:23:39)

That's a very open-minded view. You believe even greater chip companies are ahead.

Zhibin Xiao (Guest) (00:23:45)

Absolutely.

Uncle Moon (Host) (00:23:46)

After seeing so many companies and chip generations, you still see room for breakthroughs.

Zhibin Xiao (Guest) (00:23:57)

Yes. Companies like that might be the only ones able to survive in the future. Existing companies have established ways of working. Those are hard to reinvent.

Uncle Moon (Host) (00:24:11)

It's hard to disrupt yourself.

Zhibin Xiao (Guest) (00:24:11)

Exactly.

Uncle Moon (Host) (00:24:13)

So there are still plenty of opportunities?

Zhibin Xiao (Guest) (00:24:13)

Yes.

Uncle Moon (Host) (00:24:16)

The next generation should go for it?

Zhibin Xiao (Guest) (00:24:16)

Absolutely.

Uncle Moon (Host) (00:24:19) †

That's really inspiring. Will this inspire viewers to design chips?

Zhibin Xiao (Guest) (00:24:22) †

Go for it!

Uncle Moon (Host) (00:24:25)

You touched on so much. I used to work on video delivery over networks, but I'd never thought about that pipeline from a chip designer's perspective. Friends at YouTube have told me how much work goes into cutting video codec costs. At that scale, you need a lot of custom hardware.

Zhibin Xiao (Guest) (00:24:46)

Exactly. The demand is there. And the total cost of ownership makes the investment worthwhile.

Uncle Moon (Host) (00:24:52)

They're the world's biggest video service.

Zhibin Xiao (Guest) (00:24:52)

Exactly.

FPGAs, EDA, the supply chain, and RISC-V

Uncle Moon (Host) (00:24:55)

Another thing: during my PhD, I heard about FPGAs all the time. You mentioned them too, but they seem less prominent lately. Do you feel that way too? I do.

Zhibin Xiao (Guest) (00:25:08)

FPGAs aren't the mainstream choice for AI inference in data centers anymore. But they're still widely used for chip verification and emulation. They also remain useful where you need a high degree of programmability. For example, interconnects between chips where the protocol keeps changing. For a fixed workload, they may not be the most efficient implementation. But they still serve many specialized, fragmented use cases.

Uncle Moon (Host) (00:25:51)

I see. You mentioned architects, microarchitects, and verification engineers. We often hear that the US has the most advanced chip technology, with companies like NVIDIA and Apple excelling at architecture and design, while outsourcing manufacturing to TSMC. Is it fair to say the US leads in design and TSMC leads in manufacturing?

Zhibin Xiao (Guest) (00:26:20)

I'd say so. The US has strengths in IP and EDA, as well as architecture and design. RISC-V originated in the US. Arm is British, of course. There's also x86. That's instruction-set IP. For EDA, the three big names are Synopsys, Cadence, and Mentor Graphics. Without those companies, designing chips, including AI chips, would be very difficult.

Uncle Moon (Host) (00:26:53)

What does EDA stand for?

Zhibin Xiao (Guest) (00:26:55)

Electronic design automation.

Uncle Moon (Host) (00:26:58)

What does it actually do?

Zhibin Xiao (Guest) (00:27:00)

At every stage I described, engineers work with EDA tools.

Uncle Moon (Host) (00:27:07)

So, a specialized tool, like MATLAB?

Zhibin Xiao (Guest) (00:27:09)

Yes, for electronics. There are fewer tools for architecture design, but Synopsys has PAU, or Platform Architect Ultra. Cadence and Synopsys offer synthesis tools that turn RTL into a netlist. For RTL verification, there are simulation tools like VCS. Physical design has place-and-route tools. There are tools for timing checks and formal verification, as well as package design. Each engineering specialty has tools to support its work.

Uncle Moon (Host) (00:27:48)

Are those tools expensive?

Zhibin Xiao (Guest) (00:27:50)

Very. And only a few companies offer some of them.

Uncle Moon (Host) (00:27:56)

Is that because they've accumulated so much knowledge and intellectual property?

Zhibin Xiao (Guest) (00:28:03)

Yes. Over the years, the EDA industry has seen extensive mergers and acquisitions. Many small startups were acquired and consolidated into a few large companies. Second, back-end physical design tools are developed together with foundries. The physical design tools and process nodes are developed together. That requires close cooperation with the foundries. So there's accumulated capital and expertise from all those acquisitions, plus the experience gained by working with manufacturers.

Uncle Moon (Host) (00:28:59)

There's so much depth to this industry. Are other countries, including China, working on these things too?

Zhibin Xiao (Guest) (00:29:06)

Yes. The semiconductor supply chain is incredibly long. There are logic chip fabs and memory fabs.

Uncle Moon (Host) (00:29:19)

What is a foundry?

Zhibin Xiao (Guest) (00:29:20)

A factory that manufactures silicon chips. TSMC and SMIC, for example, are foundries. China also has CXMT and YMTC, with companies making DRAM and NAND. There are also many Chinese EDA and FPGA companies. The US and China have the most complete supply chains. Other countries have key strengths: ASML in the Netherlands makes lithography machines. Japan has Tokyo Electron in semiconductor equipment, plus materials companies that are indispensable to the industry. The UK has Arm and its IP. So the US and China cover the widest range of categories, Japan and Korea have their own specialties, in fabrication, equipment, and materials.

Uncle Moon (Host) (00:30:20)

You know all of this inside out. It's as if you have the industry's whole history and map of companies in your head. You also mentioned RISC-V.

Zhibin Xiao (Guest) (00:30:27)

Right.

Uncle Moon (Host) (00:30:30)

I follow Berkeley's work. They're great at open source, with projects like vLLM and SGLang.

Zhibin Xiao (Guest) (00:30:38)

Right.

Uncle Moon (Host) (00:30:38)

I was surprised to learn RISC-V is an open instruction set too. Why did they decide to create an open instruction set?

Zhibin Xiao (Guest) (00:30:46)

I've spoken with some of the people who started RISC-V. They wanted to research and develop architectures using Arm. But Arm requires a license. It was also geared toward mobile applications. It was expensive and didn't offer enough freedom for academic research. So they decided to build an open ecosystem around a new instruction set. Starting fresh has advantages. You can design it for extensibility from the beginning, without carrying legacy compatibility requirements into new workloads. RISC-V is already widely used. Many processors use its instruction set. Even NVIDIA uses RISC-V extensively for controllers inside its GPUs.

Uncle Moon (Host) (00:31:51)

Wow. I had no idea. I do know there's a big community in China and Silicon Valley, with an industry alliance.

Zhibin Xiao (Guest) (00:31:58)

Yes. There's the RISC-V Summit and the RISC-V organization. Many RISC-V startup founders, in China and abroad, are good friends of ours. In the US, there's SiFive and other RISC-V companies. In Taiwan, Andes develops RISC-V IP. In China, many Alibaba alumni have founded RISC-V companies. We know quite a few.

Uncle Moon (Host) (00:32:30)

It sounds like a small world. Everyone knows everyone, at least indirectly.

Zhibin Xiao (Guest) (00:32:34)

Yes. The industry is closely connected through its supply chain. Even with geopolitical tensions between the US and China, there's still a great deal of collaboration around open source.

Uncle Moon (Host) (00:32:52)

That's wonderful. At chip industry events and academic conferences, I see so many Chinese faces. Everyone seems connected through their universities.

Zhibin Xiao (Guest) (00:33:01)

Yes.

Process nodes, power, and on-chip memory

Uncle Moon (Host) (00:33:04)

Let's turn to the process nodes you mentioned. We hear about 2, 3, 5, and 7 nanometers all the time. What do those numbers mean?

Zhibin Xiao (Guest) (00:33:16)

Originally, those nanometer labels referred to a real physical process dimension. That was roughly before the 65-nanometer era. All chips are made of transistors. Most have three terminals: gate, source, and drain. The node number used to refer to the gate length, before around 65 nanometers. After that, it became more of a marketing term. It represents a generation of technology, not one feature or physical dimension. It might involve a new transistor structure, such as FinFET, or a new metal pitch. Those wires stacked above the silicon are metal layers, using copper or other metal interconnects. At the lowest metal layer, the spacing of the interconnects is called the M0 pitch. That's another process parameter. Then there's standard-cell size, and SRAM density. SRAM is a type of on-chip memory. Together, these parameters define a generation of technology. A manufacturer calls it 7, 5, or 12 nanometers. But no single dimension matches that label. And one vendor's 7-nanometer process isn't directly comparable with another's.

Zhibin Xiao (Guest) (00:35:08)

What you actually want to compare is the standard cell, a basic logic element, such as an AND or OR gate, and its power, performance, and area. Or you compare SRAM density and cell size. You can also compare PPA for a particular design. Two processes both called 7 nanometers aren't necessarily comparable.

Uncle Moon (Host) (00:35:36)

As a non-expert, I still can't follow all of those terms, but it certainly sounds impressive. When a process moves from 7 to 5 nanometers, or from 5 to 3, what actually improves? What problems does it solve, and what doesn't it solve?

Zhibin Xiao (Guest) (00:35:59)

There's a counterintuitive point here. People assume going from 7 to 5 or 3 nanometers makes chips faster. But the main benefit isn't speed. It's power consumption. At the same performance, power use might be cut in half. From one generation to the next, the power savings can outweigh the speed gains. For example, TSMC's move from N3 to N2, or 3 to 2 nanometers, might save 10 to 15 percent in power, while improving speed by only 3 to 5 percent. So the key improvement is often lower power at the same level of performance.

Uncle Moon (Host) (00:36:56)

That's the first time I've heard it explained that way.

Zhibin Xiao (Guest) (00:36:58)

And when power consumption falls, the same rack can hold more compute. You increase compute density.

Uncle Moon (Host) (00:37:08)

Because there's less heat to dissipate.

Zhibin Xiao (Guest) (00:37:08)

Exactly, less heat. Earlier, before the 5- and 7-nanometer era, power and area followed what's called Dennard scaling. The chip's supply voltage could fall as its dimensions shrank. But transistors have a voltage floor. You can't keep reducing it indefinitely. A chip's power consumption has two parts: static and dynamic power. Think of a logic circuit as a gate. Static power is consumed while the gate stays in a steady on or off state. Dynamic power is consumed when it switches between those states. At 12 nanometers, or 16 nanometers and above, dynamic power dominated. Since then, static power and leakage have become more significant. So the balance of power... This gets a bit complicated.

Uncle Moon (Host) (00:38:24)

That's okay. We'll do our best to follow. Great explanation. I can tell you're making it as accessible as possible. You can rewatch it and let it sink in.

Zhibin Xiao (Guest) (00:38:36)

Yes. The chip industry's technology stack is incredibly deep, with so much terminology. I'm trying to explain it, but the main takeaway is this: moving from 7 to 2 nanometers is mainly about reducing power consumption. So a 2-nanometer chip might use a fraction of the power of a 7-nanometer chip. With less heat, you can fit more compute into a rack, and get more processing power out of it.

Uncle Moon (Host) (00:39:16)

Keeping it together also reduces data movement.

Zhibin Xiao (Guest) (00:39:16)

Exactly. But there's much more to it if we dig into the details. Take AI chips, for example. Roughly half is logic and half is SRAM, a form of on-chip memory. SRAM scaling stalled as we moved through 7 and 5 nanometers toward 3. Going to a smaller process no longer necessarily shrinks the SRAM cell. That's something many people aren't aware of.

Uncle Moon (Host) (00:39:49)

I didn't know.

Zhibin Xiao (Guest) (00:39:49)

Right. The logic portion can still keep shrinking. So here's the question: what does that mean?

Uncle Moon (Host) (00:39:58)

What does it mean?

Zhibin Xiao (Guest) (00:39:58)

This is where it gets interesting. Back to AI models: serving has two stages, prefill and decode. Smaller processes help with compute-intensive work. But when memory dominates, the workload is memory-bound. Groq and Cerebras, for example, build AI chips around on-chip SRAM. If SRAM stops scaling, they have to rethink parts of their chip design philosophy.

Uncle Moon (Host) (00:40:39)

So they have to find other approaches.

Zhibin Xiao (Guest) (00:40:39)

Yes. The balance of resources in the architecture may have to change. Or they might explore things like 3D stacking.

From prompts to tokens: scheduling, prefill, and decoding

Uncle Moon (Host) (00:40:51)

That's getting pretty deep. Let's see if we can unpack it as we go. Before diving deeper into how chips and AI interact, let's cover the basics. Suppose I'm running a large model in a data center. A user sends the model a prompt.

Zhibin Xiao (Guest) (00:41:06)

Right.

Uncle Moon (Host) (00:41:10)

How do the chips in that data center work together to generate a token, and then keep generating more?

Zhibin Xiao (Guest) (00:41:18)

That's a great question. Let's walk through it. You type something and create a prompt. Your device sends it over the network to the service's gateway. On the server side, the first thing it reaches is a scheduler. Your input or question enters a queue and waits to be processed. The scheduler places it in a queue based on available GPUs and compute resources. It decides what happens next and where the request should go: which server, or which GPU. If the question needs additional context, there may also be a RAG step. That retrieves relevant history or context from a database or disk. If this is an ongoing conversation rather than one from several days ago, the context might already be in GPU memory. That context is combined with the new question to form the input prompt. The assembled context is then sent to the model. What does "sent to the model" mean in practice?

Zhibin Xiao (Guest) (00:42:55)

A runtime, a piece of software, is waiting for the input. Once the scheduler sends it the context, the runtime loads the model and begins prefill. It feeds the entire context into the model. The runtime loads the model into the GPU from memory or another location. It can also batch the input with other requests, processing many users' requests in parallel. That's the prefill stage. It's compute-intensive and parallelizable. There are no sequential dependencies between output tokens at this stage. So performance is mainly determined by the GPU's compute capability. After that, you get something important called the KV cache.

Uncle Moon (Host) (00:43:57)

Yes. Check out episode two, where Jincheng discusses his KV-cache company.

Zhibin Xiao (Guest) (00:44:05)

The KV cache is stored somewhere, perhaps on the GPU. If it's too large, it may go to storage, such as flash, or an SSD. Once you have that cache, the decode stage begins, and the model starts producing its answer. It generates one token at a time. That's decoding. This stage is trickier. It's a memory-bound workload. The cache has to be read into the GPU, and the model weights have to be accessed just to produce a single token. Generating that token adds another entry to the KV cache. That entry is stored with the existing cache. Then the updated cache and model are read again to produce the next token. Think about how much data movement that involves. That's why data movement and memory bandwidth become bottlenecks. There's another issue: the interconnect. Today's models, including open models, are enormous. DeepSeek V4 has reached 1.6 trillion parameters, and Kimi K3 has 2.3 trillion. At that size, even inference may need to be distributed.

Zhibin Xiao (Guest) (00:45:43)

Model data and KV caches have to travel between GPUs or servers. That requires communication over the interconnect. Communication time and computation time have to be balanced, so that the pipeline doesn't stall waiting for data. You don't want the slowest stage to hold everything up. There's a huge amount of runtime work here.

Uncle Moon (Host) (00:46:11)

That's what many AI infrastructure companies do. Every improvement can squeeze out more performance or save GPU memory.

Zhibin Xiao (Guest) (00:46:24)

Exactly. There's enormous room for optimization. NVIDIA, for example, says that on the B200, with models like OpenAI's open 120-billion-parameter model, two months of software optimization improved performance fivefold.

Uncle Moon (Host) (00:46:44)

And didn't OpenAI say something similar yesterday? They used GPT-5.6 to optimize its own inference and got a substantial improvement.

Zhibin Xiao (Guest) (00:46:56)

Yes. Optimization happens at many levels. There's kernel optimization, runtime organization and scheduling, model compilation, and resource scheduling. To optimize well, you first need the right abstractions. Once you have those, AI can do a lot of the work.

Simulation, profiling, and automated optimization

Uncle Moon (Host) (00:47:22)

What does that mean for optimization engineers' careers?

Zhibin Xiao (Guest) (00:47:28)

That's part of why I started this company. We want to automate the whole process, but some pieces are still missing. First, we built an AI system simulator. We combined simulation with optimization and added automated profiling. Simulation alone can capture static behavior known before compilation. But there's also dynamic behavior and low-level GPU behavior, such as kernel performance and kernel launch overhead. We collect those measurements through profiling and use them to calibrate the simulator. So the toolkit covers simulation, compiler optimization, runtime scheduling, and profiling. Our aim is to solve the optimization problem. Today's AI often can't automate this work, or do it on virtual hardware. It needs real hardware for repeated experiments.

Uncle Moon (Host) (00:48:40)

And real hardware makes that expensive, right?

Zhibin Xiao (Guest) (00:48:42)

Exactly. You have to deploy an actual system. And it can't readily explore alternative if-else scenarios that way.

Uncle Moon (Host) (00:48:55)

But from what you're saying, as AI improves and captures all that context, it should be able to optimize automatically.

Zhibin Xiao (Guest) (00:49:05)

It can. We're already using AI agents to connect our tools into a workflow.

Uncle Moon (Host) (00:49:12) †

AI-native, right?

Zhibin Xiao (Guest) (00:49:12) †

Exactly. That's what AI-native means.

Memory demand and costs

Uncle Moon (Host) (00:49:16)

Another question about what you just described: all that compute, SRAM, DRAM, and HBM. Memory has become expensive lately. Which part of the system is driving that cost?

Zhibin Xiao (Guest) (00:49:31)

First, models are getting bigger. There's memory, such as HBM and DRAM, and storage, such as NAND-based SSDs. One driver is DRAM demand for model serving. Models keep getting bigger. Second, KV caches are growing too. Their size grows with context length and the number of concurrent requests. That drives DRAM demand. HBM is a type of DRAM too. Then there's all the data we're generating. It has to be stored somewhere. Training also produces checkpoints. The amount of data keeps growing. That includes AI-generated content and intermediate data like KV caches, which may need to be offloaded to storage. That's why memory and storage are getting more expensive.

Uncle Moon (Host) (00:50:40)

Though prices have dropped sharply lately. Let's leave that discussion aside.

Zhibin Xiao (Guest) (00:50:45)

Investing is interesting. It's related to the industry, but it has its own dynamics, including capital flows.

Uncle Moon (Host) (00:50:54)

None of this is investment advice. Use your own judgment.

Groq, Cerebras, and SambaNova

Uncle Moon (Host) (00:50:57)

You mentioned data movement, chips, bandwidth, and all those bottlenecks. NVIDIA recently acquired that company founded by people from the TPU team. They make the LPU, right?

Zhibin Xiao (Guest) (00:51:12)

Right.

Uncle Moon (Host) (00:51:14)

What problem is that meant to solve?

Zhibin Xiao (Guest) (00:51:16)

GPUs aren't naturally suited to this decoding workload.

Uncle Moon (Host) (00:51:25)

That makes sense. They seem better suited to training.

Zhibin Xiao (Guest) (00:51:28)

A GPU is essentially a high-throughput streaming processor.

Uncle Moon (Host) (00:51:34)

It's great at parallel processing.

Zhibin Xiao (Guest) (00:51:34)

Exactly. That makes it very good for prefill. It's less well suited to decoding. The balance of compute and memory resources isn't right. Much of the GPU is idle while data moves back and forth through HBM. Why is Groq better at decoding? Its architecture resembles the manycore DSP I worked on during my PhD. The task's data stays on-chip. A dataflow pipeline carries the work across different chips, or rather, processor cores. Software scheduling keeps that pipeline flowing, so data doesn't constantly travel on- and off-chip. Put simply, the data stays inside the chip. GPUs aren't naturally built for that. But people are trying to reproduce those dataflow properties on GPUs. Optimizations like FlashAttention are examples. They aim to keep data inside the GPU and minimize movement. Ultimately, though, a GPU isn't a dataflow machine. So NVIDIA needs something like Groq's dataflow architecture to complement it. I was just at AMD's developer event, Advancing AI.

Zhibin Xiao (Guest) (00:53:14)

AMD is working with Cerebras, which has a similar kind of architecture, on heterogeneous inference clusters for prefill and decode.

Uncle Moon (Host) (00:53:24)

How does Cerebras differ from Groq?

Zhibin Xiao (Guest) (00:53:28)

At the logical architecture level, they're very similar. Both use on-chip SRAM and dataflow to keep data inside the chip. You map the task onto the chip without relying on external HBM or DRAM. What makes Cerebras different is wafer-scale integration. They've done some very innovative engineering. Normally, you manufacture chips across a wafer, cut it into individual dies, and package them. Then system-level interconnects connect the packages. Cerebras says: skip that. Package the entire wafer. The interconnections between the chips are already built into the wafer. You don't cut and package each chip separately. That's wafer-scale integration. One benefit is that you avoid separate packaging between chips, and the board-level connections outside those packages. The dies communicate directly across the wafer. But that creates engineering challenges, such as power delivery. An individual die can be powered from its edges. A whole wafer doesn't have enough edge access. It needs vertical power delivery.

Uncle Moon (Host) (00:55:05)

Is that because it's too thin?

Zhibin Xiao (Guest) (00:55:05)

Well...

Uncle Moon (Host) (00:55:07)

I thought the problem was how thin it is.

Zhibin Xiao (Guest) (00:55:09)

No. A chip normally gets power through pins around its edges. With a full wafer, that would only reach the chips around the perimeter. So you have to supply power vertically. Second, the wafer is large, so cooling is a major challenge. Third, there's manufacturing yield. Normally, after cutting the wafer, you discard defective dies and package only the good ones. But an intact wafer inevitably contains some defects. You need an interconnect that can route around those defective areas. So there are plenty of engineering obstacles. Personally, I think Cerebras's wafer-scale technology is very cool. But wafer-scale isn't the only way to solve the problem.

Uncle Moon (Host) (00:56:12)

Right. Tempted to start another company to tackle it?

Zhibin Xiao (Guest) (00:56:16)

There are other approaches, like 3D stacking and high-speed interconnects.

Uncle Moon (Host) (00:56:25)

So you're not especially bullish on Cerebras?

Zhibin Xiao (Guest) (00:56:29)

I think it has real strengths. A chip company can succeed if it finds a use case with enough demand. And those technologies are highly innovative. Take vertical power delivery. Many data-center chips are now moving in that direction. Cerebras was ahead in several areas. They might come up with something new.

Uncle Moon (Host) (00:56:58)

That's true. You never know. Innovation often comes from trying disruptive approaches.

Zhibin Xiao (Guest) (00:57:00)

Exactly. I think it's a remarkable company to have attempted that design at the time. They also needed TSMC to work closely with them on manufacturing. That can't have been easy.

Uncle Moon (Host) (00:57:20)

Didn't reports say Sam Altman personally invested in it early on?

Zhibin Xiao (Guest) (00:57:28)

Yes. They recognized the same opportunity. It's similar to the logic behind NVIDIA acquiring Groq. Cerebras's architecture is well suited to the decode stage of large models.

Uncle Moon (Host) (00:57:45)

These newer chip companies have released some striking demos. Decoding is so fast that the answer appears almost instantly.

Zhibin Xiao (Guest) (00:57:50)

Yes.

Uncle Moon (Host) (00:57:53)

Yes. It's impressive. The three companies I know in this space are Groq, Cerebras, and SambaNova. What does SambaNova do?

Zhibin Xiao (Guest) (00:58:02)

Several of its co-founders are former Oracle colleagues of mine. A lot of people went there after Oracle. Their CEO is a good friend too. They wanted to build a company using dataflow architecture for AI computation. Originally, the idea also included database workloads. That part attracted less demand, so they don't talk about it as much now. The architecture is also less aggressive. It relies heavily on SRAM, but not exclusively. It also uses DDR, and later HBM. It's a compromise. With Groq or Cerebras, once a model no longer fits, you need more chips. SambaNova takes a middle path: dataflow, but with DRAM and HBM as well.

Uncle Moon (Host) (00:59:08)

So all three have their strengths.

Zhibin Xiao (Guest) (00:59:08)

Exactly.

How much of a hardware gap software can close

Uncle Moon (Host) (00:59:13)

With all these chips, process nodes, and technologies, how much can software, algorithms, data handling, and frameworks like SGLang make up for a hardware disadvantage? Say one system uses chips made on a 2- or 5-nanometer process, and runs a large model, while another is limited to 7 or 12 nanometers. How much of that gap can software close?

Zhibin Xiao (Guest) (00:59:45)

First, we need to understand the gap between 7 and 2 nanometers. There's transistor power consumption at a given performance level, and compute density. Then there's the interconnect, including SerDes I/O speed. I'd put the raw performance gap at roughly two to three times. That's the underlying physical difference. But remember, SRAM stopped scaling between 5 and 3 nanometers, so that reduces part of the benefit. On the software side, you can separate prefill and decode, use low-precision quantization, speculative decoding, and multi-token prediction, and improve runtime scheduling. Those changes can recover roughly a two- to threefold improvement. There's also utilization. Software optimization can raise it from an industry average of 30 to 50 percent to 80 or 90 percent. So software can close the gap, but there's a caveat. If I have 2-nanometer chips and you have 7-nanometer chips, I can optimize my software too. An optimized older chip can close the gap with a newer chip that's poorly optimized.

Uncle Moon (Host) (01:01:42)

The good news is that whatever hardware you have, there's room to improve.

Zhibin Xiao (Guest) (01:01:48)

Exactly. But the downside is, everyone has room to improve, including the 2-nanometer system. Both sides can keep moving forward. So it's hard to make a simple comparison. Some companies don't need to optimize software as deeply because their chips advance so fast. Software development costs money too. They may not need such detailed system-level co-design. But if your process technology is constrained, you need to do more of that work. Finer optimization can bring greater benefits. But it also costs more engineering time and resources.

Uncle Moon (Host) (01:02:26)

That helps explain why constrained companies innovate.

Zhibin Xiao (Guest) (01:02:33)

Exactly.

NVIDIA's execution and supply chain

Uncle Moon (Host) (01:02:35)

Hearing how much software can recover, it sounds like NVIDIA's biggest advantage is training. More companies seem to be competing for inference.

Zhibin Xiao (Guest) (01:02:46)

That's fair. As model workloads stabilize, companies think they can build custom inference chips.

Uncle Moon (Host) (01:03:00)

Like video's boom spawning dedicated video chips.

Zhibin Xiao (Guest) (01:03:02)

Exactly. But NVIDIA has the best execution in the chip industry.

Uncle Moon (Host) (01:03:10)

That's different from how many people see it. In software and AI, people often think NVIDIA has it easy.

Zhibin Xiao (Guest) (01:03:17)

But its execution in chip development is exceptional. It moves very quickly from one generation to the next, and makes aggressive architectural changes, almost like a startup. Take support for twofold sparsity in its GPUs. They decide to do it, and ship it. Other companies are still writing papers instead of implementing it. They're also quick to optimize for particular use cases. Once they get feedback from model developers, they can evolve the architecture at startup speed.

Uncle Moon (Host) (01:03:59)

Wow.

Zhibin Xiao (Guest) (01:03:59)

The decision to acquire Groq also happened very quickly. Think about it: buying a company with an incompatible software stack, one that might compete with you, when you could build it yourself. Founder-led companies can make decisions very quickly. That's one advantage. Second, the supply chain. Jensen has said recently that control over the supply chain is a major advantage. You might design a great inference chip for a particular workload, but you may not be able to secure HBM or TSMC capacity. And your engineering team may not iterate as fast as NVIDIA's.

Uncle Moon (Host) (01:04:48)

Your perspective as a chip architect is so different from ours. Software and model developers tend to see NVIDIA's software work as slow and incremental. I hadn't realized how aggressive its hardware development is.

Zhibin Xiao (Guest) (01:05:04)

Exactly. From a software perspective, NVIDIA can look slow. Very slow. From a chip designer's perspective, its execution is the best in the industry.

Uncle Moon (Host) (01:05:18)

That makes NVIDIA's hardware moat sound almost bottomless.

Zhibin Xiao (Guest) (01:05:23)

As long as Jensen keeps leading the company, I think that advantage will continue.

Uncle Moon (Host) (01:05:31)

That's formidable. You really need to talk to insiders to understand these things. You're such a great storyteller. This is going to be a fascinating episode.

Inference costs and physical infrastructure

Uncle Moon (Host) (01:05:44)

Here's another question. Many of today's AI bottlenecks involve inference. Claude Code went down again today. Demand is clearly enormous.

Zhibin Xiao (Guest) (01:05:48)

Exactly.

Uncle Moon (Host) (01:05:54)

Is inference most constrained by compute chips, HBM, networking and interconnects, power, or something else? What's in shortest supply?

Zhibin Xiao (Guest) (01:06:05)

I'd say compute capacity. But the key bottleneck is the ability to keep optimizing the system, and keep bringing down the cost per token. Models are proliferating, chip architectures vary, and systems are becoming heterogeneous. Add multimodal models and agent infrastructure, and there's huge room for improvement. We need fast system-level co-design and faster iteration, so we can reduce the cost per token. Keep driving that cost down until AI is as cheap and accessible as tap water.

Uncle Moon (Host) (01:06:56)

Exactly. AI should be a utility, like water or electricity.

Zhibin Xiao (Guest) (01:07:06)

For those of us in the industry, that isn't just a possibility. It's going to happen.

Uncle Moon (Host) (01:07:11)

Right. It's hard to imagine tokens staying this expensive forever.

Zhibin Xiao (Guest) (01:07:15)

The cost will definitely come down. The semiconductor industry keeps innovating. There's KV-cache optimization and distributed inference, and beyond software, a new generation of chip companies is emerging. Memory is a bottleneck, and device companies are developing new solutions. There are new optical interconnect solutions too. Where there's a bottleneck, engineers will find ways to address it. Especially now that AI can help. But manufacturing capacity and infrastructure have physical limits and their own timelines. Take the power shortage in the US. Building a power plant or expanding the grid can take five years or more. Building a foundry has its own timeline too. These constraints come down to how fast things can happen in the physical world. Even the annual supply of lithography machines is a bottleneck.

Uncle Moon (Host) (01:08:23)

Right. Engineers will solve everything they can with the resources available. But manufacturing capacity and production speed impose limits, so bringing token costs down still takes time.

Zhibin Xiao (Guest) (01:08:42)

Exactly. That's why, when the physical world is constrained, software has to help. Right now, a huge amount of compute is wasted because of software inefficiency. Many GPUs have very low utilization. And many hardware features go underused, because models change so fast and hardware is so varied. People haven't had time to optimize everything carefully. Cost pressures and physical constraints force them to become more efficient, and to automate that optimization. We want to turn our chip and systems expertise into something an AI agent can use, so it can keep optimizing automatically.

Uncle Moon (Host) (01:09:39)

I really believe an AI-native company like yours has a bright future.

Zhibin Xiao (Guest) (01:09:45)

Thank you. I appreciate that.

Oracle, compression architecture, and career choices

Uncle Moon (Host) (01:09:47)

We've never started an interview this technically. Over an hour of deep technical discussion! Let's switch gears and return to your personal story. You finished your PhD in 2012.

Zhibin Xiao (Guest) (01:09:59)

Right.

Uncle Moon (Host) (01:10:02)

You could have chosen academia, a chip company, or a software company. Why Oracle? What attracted you most?

Zhibin Xiao (Guest) (01:10:11)

A major reason was that a new team was working on something new. It was a landmark hardware-software co-design project: accelerating database operations inside a CPU. It involved co-designing the chip architecture, working across hardware and software, and iterating with the Oracle Database runtime. It was a complete, innovative project. In the chip industry, opportunities to work on something end to end used to be rare. Architect roles usually went to very experienced people. A new engineer might only design one block or module, or start in verification and gradually work their way up. But Oracle had just acquired Sun Microsystems and wanted a new accelerator for its own business. That created an opening. Our team was called Software in Silicon. The idea was to implement software functionality in hardware. It was highly innovative, and I saw it as a great opportunity.

Uncle Moon (Host) (01:11:44)

That was a rare opportunity. Viewers often ask me which company to join and how to make that choice. Personally, I think your approach makes sense: choose a role that lets you see the bigger picture. When you're young and have just graduated, you don't want to confine yourself to something narrow, dull, and unchallenging.

Zhibin Xiao (Guest) (01:12:13)

Yes. While you're young, seek out work that lets you understand the whole system. Say you can join one of today's leading AI infrastructure startups, or a big company working on traditional databases. I'd recommend the AI infrastructure company, where you can learn the whole pipeline. Don't choose a big company just for an impressive title if the work is stuck in the distant past.

Uncle Moon (Host) (01:12:41)

Exactly.

Zhibin Xiao (Guest) (01:12:45)

In a mature workflow, you may just be patching and maintaining other people's work. Of course, that job may be easier, and people have different priorities. Some want stability or need to sort out their immigration status. There's nothing wrong with that choice. But if those concerns are covered, innovative work with a broad view is a great choice. That's why I've kept choosing new, challenging projects throughout my career.

Uncle Moon (Host) (01:13:21)

You seem so happy. Ever since I met you, you've always had a smile on your face. Your work seems to give you energy. I really like that attitude. And you're thoughtful too. You've been working for nearly 20 years, yet you still remember that new graduates have immigration and family concerns. You're both positive and considerate. It's lovely to see.

Zhibin Xiao (Guest) (01:13:52)

Thank you.

Uncle Moon (Host) (01:13:54)

You've described Oracle's project as innovative, accelerating database queries. What was the most interesting or consequential decision you made there?

Zhibin Xiao (Guest) (01:14:06)

In chip and system design, as systems researchers know, it's all trade-offs. Nothing is perfect. There's no free lunch. What do you choose or give up?

Uncle Moon (Host) (01:14:14)

Exactly. Which decision helped you grow the most, or made the biggest difference to the company?

Zhibin Xiao (Guest) (01:14:22)

DAX had two main parts. One handled memory access, fetching the data. The other was the query processor, the database acceleration or compute unit. One of our biggest decisions was choosing a compression format for the database. The compressed data sat in memory. We'd read it in, decompress it in real time, and compute on it immediately. We extracted only the data needed for the operation, then sent the results back to memory. Choosing compressed computation was a major decision. So was choosing the compression algorithm. There are trade-offs. Once you've committed to an algorithm, a change in the data type or workload can hurt compression efficiency. That's a significant risk. I was responsible for the compression architecture, algorithm, and microarchitecture. Oracle had a compression algorithm called OZIP, or Oracle ZIP. We did a lot of work on the software side to adapt it, evaluate compression efficiency, and compare possible implementations. We ultimately chose a table-lookup approach.

Zhibin Xiao (Guest) (01:16:06)

For our query workload, the database didn't change very much. We could scan it first and build a compression codeword table. That table could be preloaded onto the chip before execution. We loaded it into the accelerator. When a query arrived, we looked up the codewords, decompressed the corresponding data, and then performed the query operations. If the data changed, we could scan the database again and generate a new table. That new table would then be used for decompression. So although the choice involved risk, we had a way to reduce it. It was fascinating work.

Uncle Moon (Host) (01:17:12)

I can't follow every detail, but I can tell you were doing something different. Over those five years at Oracle, you clearly enjoyed working on an innovative project. What were Oracle's greatest strengths, and where did it fall short?

Zhibin Xiao (Guest) (01:17:30)

After acquiring Sun, Oracle had many outstanding engineers and architects.

Uncle Moon (Host) (01:17:39)

For viewers who don't know, Sun was the company behind Java and MySQL.

Zhibin Xiao (Guest) (01:17:46)

Sun Microsystems.

Uncle Moon (Host) (01:17:46)

Exactly.

Zhibin Xiao (Guest) (01:17:47)

In the 1990s, it was a star company, much like NVIDIA or Google today. It also supported many open-source projects. Oracle bought it for around seven billion dollars. That was incredibly cheap. What came with the acquisition? The SPARC CPU division, Solaris, Java, and MySQL, whole systems, including storage systems. There was a lot. From today's perspective, seven-something billion dollars was a bargain. The company had excellent vertical integration. We had CPU design, databases, and the Solaris operating system, plus runtime scheduling. You could find an expert on practically anything. That vertically integrated organization was a real strength. But the project ultimately wasn't especially successful. Why? SPARC had its own instruction set, incompatible with x86. Meanwhile, x86 servers running Linux became the data-center mainstream. Before that, it had been SPARC and Solaris. Solaris didn't embrace open source, and the ecosystem failed to flourish. Fewer developers built applications for SPARC. The ecosystem weakened, development slowed, and the project struggled.

Zhibin Xiao (Guest) (01:19:37)

Eventually, Oracle shut down the SPARC division, including our teams.

What open source means for people and companies

Uncle Moon (Host) (01:19:45)

Let's pause before the layoffs. You just mentioned open source. A big story in Silicon Valley lately is Jensen's call to support open source, Open model weights.

Zhibin Xiao (Guest) (01:19:58)

Right.

Uncle Moon (Host) (01:19:59)

More companies are joining that effort. I'm a strong supporter of open source too.

Zhibin Xiao (Guest) (01:20:04)

Right.

Uncle Moon (Host) (01:20:06)

Looking back over your career, what does open source mean for countries, companies, and individuals?

Zhibin Xiao (Guest) (01:20:16)

Wow, that's a big question.

Uncle Moon (Host) (01:20:16)

It is. It's so important that I had to ask, even though we hadn't planned it.

Zhibin Xiao (Guest) (01:20:19)

Absolutely. In hardware, open source is still relatively uncommon. Open IP or instruction sets like RISC-V are unusual. Much hardware IP, including SerDes, is proprietary and requires a license. On the software side, a chip needs software and an ecosystem. Open-source software helped the internet flourish. Think of Google's influential database papers, x86 servers running open-source Linux, and open-source big-data systems. They enabled the last wave of internet growth. For the industry, I'm absolutely in favor of open source. It helps create a thriving ecosystem. It's good for individual engineers too. We can contribute to larger, more complex projects, and our skills become transferable. When we move between companies, our technical knowledge moves with us. But open-sourcing models raises another question: why do some companies hold back? They've spent enormous sums training those models. If they open-source them, how do they earn a return? Everyone else benefits.

Zhibin Xiao (Guest) (01:22:10)

For model companies, especially the frontier leaders, recovering that upfront investment after open-sourcing a model is a serious challenge. A newcomer can use open source to grow its ecosystem. But it isn't necessarily the best strategy for every company. It depends on whether you're a market leader or a challenger. There are advantages and disadvantages depending on your position. For individuals, though, open source is clearly beneficial. You can deploy your own private OpenClaw agent, use open models for all kinds of tasks, and keep control of your data. For businesses, if an open-source solution meets compliance and security needs, it can reduce costs, and give them bargaining power with model providers. At the national strategy level, it's a much bigger question. Again, it comes down to leaders and followers. If China were in America's position, it might not choose open source either. Using open source to catch up is a common strategy for challengers.

Uncle Moon (Host) (01:23:52)

That's a very balanced analysis. Open source is a complicated issue. Each group has its own concerns. But for us as individuals, the benefits are clear.

Zhibin Xiao (Guest) (01:24:00)

That's right.

Uncle Moon (Host) (01:24:06)

As an individual, I'll always support open source.

Zhibin Xiao (Guest) (01:24:06)

Yes.

From architect to product leader and CEO

Uncle Moon (Host) (01:24:09)

In just five years, from 2012 to 2017, you became a Principal Engineer at Oracle. People in Silicon Valley will appreciate how difficult that is. You start as a junior engineer, then senior, then principal. That's a very senior role.

Zhibin Xiao (Guest) (01:24:28)

I should clarify that there was some title inflation at Oracle then. Our Principal title wasn't quite the same as Principal or Fellow at NVIDIA. But it was still a good position.

Uncle Moon (Host) (01:24:43)

What helped you advance so quickly?

Zhibin Xiao (Guest) (01:24:48)

Making a real contribution to the project. The company could see my output and impact. I took on difficult, challenging work, like the hardware-software co-design for compression. I also coordinated resources across a broader scope. I wasn't just handed a specification and asked to implement a single block. I was doing more of an architect's job.

Uncle Moon (Host) (01:25:28)

So you took on architectural responsibility early, with a wider scope, and naturally grew into a bigger role.

Zhibin Xiao (Guest) (01:25:32)

Exactly. The problems you solve and the projects you take on determine that growth.

Uncle Moon (Host) (01:25:43)

After that, you joined a newly formed chip team in China. What led you to make that move?

Zhibin Xiao (Guest) (01:25:51)

At the time, hyperscalers were starting to develop custom chips. Everyone wanted to get into it. When we joined this internet company, it wasn't even clear what kind of chip to build. That was a great opportunity for me. I wasn't just an architect anymore. I was also a product manager, working out which chip made sense from a business perspective. Normally, a chip company's CTO or CEO makes that decision. We researched use cases, business needs, return on investment, and costs. Was a workload stable enough to justify a custom chip? We did a lot of that work. Second, we were building everything from scratch. We brought in design services and decided which IP to buy, worked with foundries, and signed the first partnership agreements. There was a lot to evaluate, including things outside a typical technical role. That was useful preparation for starting a company.

Uncle Moon (Host) (01:27:15)

There's a clear pattern in your choices: building from scratch and taking broader ownership. Almost like founding a startup inside a big company.

Zhibin Xiao (Guest) (01:27:27)

Exactly. AI hadn't really taken off yet. We weren't necessarily going to make an AI chip. Whenever I make a career move, I ask whether I'll learn something I haven't done before. Good pay and a wider scope help too. As a team grows inside a large company, you also learn how to work with more people. But when we joined, we were in a great position: the founding team of an internal startup.

Uncle Moon (Host) (01:28:21)

You enjoy taking a vague problem, clarifying it step by step, and solving it end to end.

Zhibin Xiao (Guest) (01:28:31)

Yes. Architecture is about finding certainty within uncertainty, defining the problem and then solving it.

Uncle Moon (Host) (01:28:43)

That's beautifully put. There's a lot to learn from how you've shaped your career. I'm similar. I enjoy defining a problem and building something from zero to one. I love zero to one, less so one to a hundred. Those opportunities can be rare.

Zhibin Xiao (Guest) (01:29:00)

Building AI chips at an internet company is unusual. The culture differs from a traditional chip company. The processes don't exist yet. You get to build them from a blank page. That experience helped me a lot when I started a company later.

Uncle Moon (Host) (01:29:22)

You then became involved in two startups.

Zhibin Xiao (Guest) (01:29:22)

Right.

Uncle Moon (Host) (01:29:27)

At the first, you were Chief Architect. At ZFlow.AI, you chose to be CEO. Why make that change?

Zhibin Xiao (Guest) (01:29:39)

At my previous company, I was still an architect. That was a continuation of my earlier work. But I did take on new responsibilities. Beyond chip architecture, I got involved in system software and runtime teams. I also brought in capabilities we'd developed at Alibaba, such as finding partners. But AI infrastructure software is very different from making chips. The chip business model is fairly well established. You build a chip that will be competitive in two years. Then sell chips, boards, and related software services. The route to market is relatively clear. AI infrastructure changes much faster. We've built tools and products, but how do you sell them? How do you price them? What exactly are you offering? Consulting, services, or a subscription? That's a new challenge for me. I felt I had to take it on myself. It's hard to find someone who can combine technical innovation with a new business model. So this is another learning experience.

Uncle Moon (Host) (01:31:25)

You like challenging yourself and enjoying the journey.

Zhibin Xiao (Guest) (01:31:28)

It is challenging. Selling software really is different from selling chips.

Uncle Moon (Host) (01:31:34)

Now that you've moved from Chief Architect to CEO, How much time goes to product, sales, and marketing? What about fundraising and technology?

Zhibin Xiao (Guest) (01:31:48)

I still spend a lot of time on the product and technology. Most of my time goes to business development and partners, and talking with potential customers. signing MOUs, and working out the business model together. That takes up a large share of my time. Our burn rate is relatively low. Fundraising takes some effort too, I'd rather raise funds in a concentrated window, instead of letting it consume my time indefinitely. For now, most of my attention is on the business and business development. I still follow product decisions closely and understand the technical details.

Uncle Moon (Host) (01:32:41)

So you spend time with customers, defining the product and business model. Once those are clear and you have customers, you take a month, raise the money, and get on with it.

Zhibin Xiao (Guest) (01:32:50)

Exactly. That's my approach, though I'm still learning how to be a CEO. Some people keep fundraising. More capital lets you do more. Coming from chips, I'd rather establish the business, paying customers, and clear value first. Then fundraising should follow naturally. The point of raising money is to support the business. Once there's a substantial business, I think raising money becomes easier.

Uncle Moon (Host) (01:33:29)

I've heard successful Silicon Valley founders say fundraising comes almost as a by-product. Once the business works, investors line up to help you expand, and you turn meetings down.

Zhibin Xiao (Guest) (01:33:42)

Yes. Fundraising takes a different skill set from building products and a business. Spend too much time on it, and you have less time for the business itself.

Uncle Moon (Host) (01:33:57)

Fundraising matters too. Some companies survive because their CEO keeps raising capital.

Zhibin Xiao (Guest) (01:33:59)

Exactly.

Uncle Moon (Host) (01:34:04)

And knowing how to achieve a good exit matters too.

Zhibin Xiao (Guest) (01:34:04)

Absolutely.

Uncle Moon (Host) (01:34:08)

We each have strengths.

Zhibin Xiao (Guest) (01:34:09)

I'm still learning.

Uncle Moon (Host) (01:34:09)

Always learning.

Zhibin Xiao (Guest) (01:34:09)

Right.

B300 results, simulation accuracy, and a neutral platform

Uncle Moon (Host) (01:34:13)

Your company recently published inference optimization results on the B300. You reported 1.5 times the peak throughput, and a two- to threefold improvement in tail latency. How did you achieve that?

Zhibin Xiao (Guest) (01:34:28)

We supported the model on day zero, or very soon after release, using a serving framework for DeepSeek V4 on eight B300 GPUs. We used simulation to find promising configurations, then applied them to the serving framework. For example, with eight B300s, should you separate prefill and decode? If so, how many GPUs should you assign to each stage? Simulation helped us make those decisions. We then configured the serving runtime accordingly. There was also a lot of engineering work. Making a concurrent serving runtime stable at high throughput is difficult. The open-source framework itself was still evolving. Our approach is to find good configurations quickly through simulation, then integrate them with the serving framework for a better deployment. That's the basic idea. But we spent a lot of time getting the framework stable. DeepSeek had just been released, and there were plenty of issues.

Uncle Moon (Host) (01:36:01)

I've seen people discussing exactly that on social media: tuning these open-source runtimes is incredibly complicated. Finding the best result takes a lot of time and resources. The framework developers must know that too.

Zhibin Xiao (Guest) (01:36:13)

They do.

Uncle Moon (Host) (01:36:16)

If they're optimizing their own systems, what advantage do you bring?

Zhibin Xiao (Guest) (01:36:23)

We're also exploring partnerships with the open-source ecosystem. Our simulator can provide a virtual backend for those projects. Their runtime can run in a simulated environment, so they don't need as many GPUs to do the optimization. They can optimize before hardware arrives. So I see us as complementary.

Uncle Moon (Host) (01:36:48)

In a way, this is also an EDA tool, helping people iterate and optimize.

Zhibin Xiao (Guest) (01:36:52)

In a sense, yes. Many customers come to us for architectural design-space exploration. The tool is useful for chip design, especially AI chips. When we designed chips, building a simulator was the first step. So you can think of it as an architectural EDA tool. It can still support chip design and architecture exploration, but it can also optimize data centers. Some chip companies already expose internal simulators to people building inference systems. That lets them iterate on optimization strategies faster. Our difference is that we want a neutral platform. It should support different chip architectures and systems, including combinations of NVIDIA, AMD, or other vendors' hardware. You can simulate and optimize them together for lower-cost inference or training. That's the goal: a general-purpose, vendor-neutral platform.

Uncle Moon (Host) (01:38:17)

So customers aren't tied to one vendor, and everyone can use it.

Zhibin Xiao (Guest) (01:38:17)

Exactly. The industry needs tools for exploration and tuning without requiring physical GPUs.

Uncle Moon (Host) (01:38:35)

Were those results, including 1.5 times peak throughput, simulated or measured?

Zhibin Xiao (Guest) (01:38:41)

On real hardware. We worked with a neocloud provider. They supplied the GPUs; we did the tuning.

Uncle Moon (Host) (01:38:50)

How close were the measurements to your predictions?

Zhibin Xiao (Guest) (01:38:54)

For DeepSeek, we look at prefill and decode separately. For prefill, our simulation is over 90 percent accurate against actual runs. For decode, it's roughly 80 to 90 percent. The difference comes from dynamic behavior when the runtime actually executes. We can simulate different scenarios, but matching the actual data distribution precisely is difficult. So we're approaching 90 percent simulation accuracy.

Uncle Moon (Host) (01:39:38)

By dynamic behavior, do you mean unpredictable user requests, and things like data-center temperature?

Zhibin Xiao (Guest) (01:39:45)

Yes. There's dynamic batching and concurrency, and expert scheduling in mixture-of-experts models. Different input distributions lead to different expert routing. Those dynamic behaviors are hard to simulate. We can model the distribution or feed profiling results back into the simulator. We use those measurements to calibrate the simulator's predictions. We're unusual in that we come from chip architecture and use simulation for optimization. But achieving enough simulation accuracy is a difficult problem. Making it useful for optimization requires careful abstraction. You have to abstract the system architecture and the network. Distributed inference needs network simulation. How much detail do you model in the topology, switches, and routing? Do you simulate the congestion-control algorithm? Or is a simpler model accurate enough for optimization? You need experience with complex real systems to build these abstractions effectively. It's tough.

Uncle Moon (Host) (01:41:28)

You're building a virtual data center.

Zhibin Xiao (Guest) (01:41:31)

In a sense. NVIDIA recently introduced DSX Air, an AI token factory simulator. Our concept is somewhat similar. Jensen talked about that at GTC: an AI token factory simulator.

Uncle Moon (Host) (01:41:48)

That's incredibly hard. Congestion control determines how packets are sent and received. Then there's the router and switch architecture, and even the wiring layout. All of that affects the result.

Zhibin Xiao (Guest) (01:42:01)

Expert-routing policies in MoE models affect it too. But I'm confident we can find useful abstractions. Compared with general-purpose CPU algorithms, AI workloads are predictable. That makes abstraction possible.

Agent optimization and the business model

Uncle Moon (Host) (01:42:25)

On your website, alongside AI infrastructure optimization, you also have an OpenClaw AutoTuner, and a memory manager.

Zhibin Xiao (Guest) (01:42:34)

Right.

Uncle Moon (Host) (01:42:36)

Putting those three together makes the picture much clearer. OpenClaw handles the application layer, user requests, and interactions. Then you optimize memory use and its relationship to compute. Is that the idea?

Zhibin Xiao (Guest) (01:42:54)

Yes. We're working at two levels. What we discussed earlier was model-serving optimization. OpenClaw is agent-level optimization. Based on user behavior, it chooses models and allocates memory, finding the best configuration for the agent. We already have plenty of AI chips and open-source frameworks, plus agent infrastructure like OpenClaw. But how do you configure and tune it? Every case is different. Requiring lots of engineers to tune each one is cumbersome. Autotuning model serving and agent infrastructure follows the same idea: automate optimization.

Uncle Moon (Host) (01:43:47)

Didn't the database industry have autotuning companies too?

Zhibin Xiao (Guest) (01:43:50)

Exactly. Databases have query optimizers. They decide where to run work and how to allocate memory. That can be as complex as AI, or even more complex. Database workloads have lots of unpredictable, data-dependent behavior. AI has dynamic behavior too, including MoE routing and batching, but much of the work is still regular matrix computation. As AI starts to resemble database workloads, more dynamic behavior enters the picture. Some of the optimization algorithms can be similar.

Uncle Moon (Host) (01:44:42)

Isn't it hard to commercialize? How many database optimization companies actually survived?

Zhibin Xiao (Guest) (01:44:50)

You're right. Very few database acceleration companies succeed. One reason is that the workload and business requirements change quickly. The software stack is very deep. When optimizer developers use specialized hardware, keeping up can be difficult. And database companies often already have their own optimizers. That makes it difficult to define a viable business model.

Uncle Moon (Host) (01:45:22)

Exactly. Database experts I really admire have started companies like that, and struggled commercially despite brilliant technology.

Zhibin Xiao (Guest) (01:45:37)

That's why I'm still learning. In AI inference, you either need customers who trust you and need what you offer, or you need resources, such as GPU capacity. You need a foothold on one side or the other. It's difficult to survive as a pure infrastructure or optimization company.

Uncle Moon (Host) (01:46:03)

Some apparent AI infrastructure companies are really GPU businesses.

Zhibin Xiao (Guest) (01:46:07)

Exactly. Many are GPU businesses, or they're closely tied to a major customer.

Uncle Moon (Host) (01:46:19)

Knowing that history, you must see how challenging the business could be.

Zhibin Xiao (Guest) (01:46:26)

There are real challenges, both technical and commercial. But it's an important problem, and solving it would create enormous value. The industry needs a neutral simulation platform that also does optimization. Serving runtimes need a virtual backend. There's a lot we can do.

Uncle Moon (Host) (01:46:52)

I can see how open-source projects and hardware vendors could benefit. Doing experiments through your virtual backend, without GPUs, could save time and money.

Zhibin Xiao (Guest) (01:47:06)

In a sense, we can become the architecture team for an AI infrastructure or chip company. We have both the tools and the people.

Uncle Moon (Host) (01:47:17)

That sounds like a business someone might want to acquire.

Zhibin Xiao (Guest) (01:47:21)

That's one possibility.

Uncle Moon (Host) (01:47:22)

Your description also reminds me of VMware.

Zhibin Xiao (Guest) (01:47:22)

Right.

Uncle Moon (Host) (01:47:28)

It had a difficult path to commercialization, then suddenly took off.

Zhibin Xiao (Guest) (01:47:34)

It built virtual machines.

Uncle Moon (Host) (01:47:34)

Exactly. You're building a virtual backend and those layers of abstraction. The business model may not be obvious at first, but one day, every cloud provider might need to deploy it.

Zhibin Xiao (Guest) (01:47:49)

I think Jensen already sees that, given his AI token factory simulator. AI hardware and system software have become incredibly complex. Deploying a real system just to optimize it is very expensive.

Uncle Moon (Host) (01:48:09)

Not necessarily a shortage of engineers. For many AI infrastructure companies, it's hardware.

Zhibin Xiao (Guest) (01:48:16)

I think engineers are scarce too. These systems are too complex for most new graduates to handle.

Uncle Moon (Host) (01:48:23)

Yes, the complexity is substantial. The people I've seen handle this well usually have years of industry experience with GPUs. or have learned a great deal by contributing to SGLang or vLLM. From my experience and these interviews, it's very hard to jump straight into this work.

Zhibin Xiao (Guest) (01:48:45)

That's true. Our team's experience is unusual. We're all architects. We've deployed heterogeneous systems and built GPU clouds from scratch. One colleague from my PhD group has presented at Hot Chips and spent years developing simulators. I've spent my career on hardware-software co-design for chips. In a sense, we're solving our own problem: how to optimize systems and design architectures.

Team experience, engineer pay, and future careers

Uncle Moon (Host) (01:49:23)

You're your own first customers.

Zhibin Xiao (Guest) (01:49:23)

Exactly.

Uncle Moon (Host) (01:49:27)

How did you meet? How big is the team?

Zhibin Xiao (Guest) (01:49:29)

The core team is small, just a few people. They're former colleagues and people from my research group, with shared experience.

Uncle Moon (Host) (01:49:41)

Veterans who've been through the trenches together.

Zhibin Xiao (Guest) (01:49:41)

Exactly.

Uncle Moon (Host) (01:49:44)

That's very cool. The experience and knowledge in your heads must be invaluable. For younger people who haven't been through those battles, even with AI agents, it's hard to know how to acquire that knowledge. I honestly don't know how you get that kind of experience.

Zhibin Xiao (Guest) (01:50:07)

You're right. Some of it comes from making mistakes and learning from them. Chips are expensive, so accumulated experience matters a great deal. Now inference and chips have become bottlenecks. But historically in Silicon Valley, internet companies paid more. Many friends of mine switched into software engineering. Relatively few of us stayed in semiconductors, steadily doing the work. People may not know this about chip engineers in Silicon Valley: they often earn much less than software engineers.

Uncle Moon (Host) (01:50:53)

Why is that?

Zhibin Xiao (Guest) (01:50:54)

It comes down to how the business uses capital. A chip company spends heavily on hardware and capital investment, tape-outs, EDA licenses, and IP. People are a smaller part of that cost structure. Top architects are still scarce, of course. But most engineers work with EDA tools, and their pay isn't as high. At internet companies, before AI and GPU infrastructure, most spending went to people. People were the key resource. Once you'd built the software, you could scale it almost indefinitely. That's changing now. Cloud companies are starting to resemble semiconductor companies: capital spending dominates. Meanwhile, some human work may be replaced by AI. Internet companies and hyperscalers are moving toward that more capital-intensive model. Capital expenditure is becoming the biggest investment. The costs associated with revenue scale with hardware investment, rather than simply replicating software to grow without limit.

Uncle Moon (Host) (01:52:21)

That seems right. Take the model-training pipeline: pretraining, mid-training, and reinforcement learning. Once that is built, human labor may become less valuable. Models can already help people run the entire training process. A new graduate can follow instructions from the model to operate the system.

Zhibin Xiao (Guest) (01:52:45)

The semiconductor analogy is useful: the model becomes your EDA tool, and software engineers become the people operating it. Big companies then invest more in infrastructure. We may have to accept that a small group at the top, like chip architects, will still earn very high incomes and returns. But most software engineers will work with AI tools, and compensation will change.

Uncle Moon (Host) (01:53:27)

Today's high salaries partly reflect scarce skills and the race to stay ahead. Once the industry matures, it's hard to imagine everyone paying what companies like Meta pay now.

Zhibin Xiao (Guest) (01:53:44)

Even now, those extraordinary salaries go to a very small group. Headlines can make it sound much more widespread. Think of them as the equivalent of top chip architects. People in Silicon Valley know there are only a handful of them.

Uncle Moon (Host) (01:53:57)

Exactly. Given that outlook, should today's high school students still study computer science or chip design?

Zhibin Xiao (Guest) (01:54:14)

Yes, absolutely. Everyone needs to think like an architect: connect the layers and use tools to build things. The demands on people are actually higher. You still need to learn the fundamentals. Computer science still matters. But you should also understand business and business models, and develop your own judgment and practical know-how. Bring that architectural way of thinking to your career choices and training.

Uncle Moon (Host) (01:54:53)

Right. On the other hand, if someone isn't deeply interested, should they feel less pressure to pursue it? If you love it, go for it.

Zhibin Xiao (Guest) (01:54:59)

Absolutely.

Uncle Moon (Host) (01:55:02)

But without that interest, becoming a top architect could be a painful journey.

Zhibin Xiao (Guest) (01:55:07)

True, if it isn't what you enjoy. Still, I think AI will raise productivity dramatically. You might work like a semiconductor engineer, using AI as your tool. You may not earn the very highest salary, but higher productivity should make a decent basic standard of living possible.

Uncle Moon (Host) (01:55:28)

Right. A comfortable life without worrying about the basics sounds pretty good too. Ordinary semiconductor engineers may earn less, but a decent life, perhaps with less overtime and pressure, has its appeal, doesn't it?

Zhibin Xiao (Guest) (01:55:45)

To some extent, yes. But society may become more polarized. Many people may not need to work at all.

Closing remarks

Uncle Moon (Host) (01:55:54)

I've really enjoyed this conversation, Zhibin. Much of it wasn't planned. You gave us so much to explore that I was improvising. I hope we've helped viewers understand your 20-year journey through the chip industry, and how Silicon Valley's chip industry has evolved through so many generations.

Zhibin Xiao (Guest) (01:56:21)

Yes. Thank you for inviting me, Uncle Moon. I may have spoken a little quickly. We could spend several more hours on any one of these topics.

Uncle Moon (Host) (01:56:32) †

Then come back! Let's call this an introduction. We can return and explore each chapter in more detail. Thanks, everyone! Thank you, Zhibin.

Zhibin Xiao (Guest) (01:56:41)

Thank you.

† At least one speaker change or brief response in this passage needs audio verification.