Host: Uncle Moon
Guest: Junchen Jiang

Translated in full from the Chinese subtitles. Dialogue is grouped into paragraphs, retaining the conversation and opening highlights.

Speaker labels are inferred from context and have not been checked against the audio throughout. A dagger (†) marks passages with uncertain attribution. Timestamps come from the subtitles; short exchanges within one subtitle may share a timestamp.

Contents

Highlights

Junchen Jiang (Guest) (00:00:00) †

I wouldn't even go to the selection sessions for the Special Scholarship. Ha ha. Even sitting in the audience felt like a waste of time to me.

Uncle Moon (Host) (00:00:06)

Today, we wanted to talk to one of the culprits.

Junchen Jiang (Guest) (00:00:09)

Ha ha, back then AI wasn't even a thing. Ha ha. A whole live-streaming system, built specifically for the Beijing Olympics. KV cache is gold. It's the oil of the future.

Uncle Moon (Host) (00:00:19)

We can make that bet. Then we'll see how it actually turns out.

Junchen Jiang (Guest) (00:00:23)

The time I spent in industry writing product code and doing engineering work was the most helpful for my research. As for investors, that's a pitfall a lot of professors have stumbled into.

Introduction and Rising Storage Prices

Uncle Moon (Host) (00:00:36)

Interviewing 100 fascinating people in Silicon Valley. This is Uncle Moon. Today we're joined by Junchen Jiang. Junchen's company recently raised $20 million. Junchen, please introduce yourself.

Junchen Jiang (Guest) (00:00:47)

Hi, everyone. My name is Junchen Jiang. I'm a professor in the computer science department at the University of Chicago, and I'm also the CEO and co-founder of Tensormesh.

Uncle Moon (Host) (00:00:55)

To add a little background, Junchen earned his PhD at CMU and his undergraduate degree in Tsinghua's Yao Class. I read his papers quite a long time ago, because my PhD was also in networked multimedia. Over the past two years here in Silicon Valley, I've watched him move from networked multimedia into large models, and even start a company. So I really wanted to understand his journey and see what new insights he could share with everyone.

Junchen Jiang (Guest) (00:01:16)

There are lots of stories to share.

Uncle Moon (Host) (00:01:17)

Great, we love stories. Ha ha. Before we get into those, let's start with a hot topic: storage prices have gone up enormously recently. For the past two years you've been building a company around KV cache. Cache, caching, storage—it all sounds closely connected. What do you make of this surge in storage prices?

Junchen Jiang (Guest) (00:01:36)

First, rising storage prices follow the laws of business. It's simply a mismatch between supply and demand. People need to buy more storage devices, but there isn't enough storage capacity, or enough devices. The key question, though, is whether anyone has thought about why we need so much storage in the first place. Right? That's a very interesting question. In the AI era, everybody is focused on computation and compute capacity. Nobody thinks about storage capabilities or storage capacity. But here's the interesting part: the more compute you have, the more storage you need. Perhaps people didn't recognize that at first. So now the market needs to adjust. Prices being too high is quite a normal thing. What I find really interesting is why storage has become such a big issue in the AI era. What we work on, including KV cache, is data generated by models as they compute and run.

Junchen Jiang (Guest) (00:01:36)

This kind of data is one of the main reasons storage is under such pressure now. We can get into that later, but I think we may be one of the reasons those prices are going up.

Uncle Moon (Host) (00:02:43) †

Right. That's why we wanted to talk to one of the culprits today. Okay, right, great.

Joining the Yao Class and Finding Research Interests

Uncle Moon (Host) (00:02:48)

We'll come back to that in detail. First, let's look at Junchen's earlier experiences as a student. From his official website, it looks like he graduated from Tsinghua in 2011, which would mean he took the college entrance exam in 2007, right?

Junchen Jiang (Guest) (00:03:03)

Strictly speaking, I didn't take the college entrance exam. I was admitted through a recommendation. Ha ha.

Uncle Moon (Host) (00:03:08)

Tell us more about how you got that recommendation.

Junchen Jiang (Guest) (00:03:10)

It was through a math competition. I entered Tsinghua in 2007, so the math competition was in 2006. Once you reached a certain level, you could be recommended for admission.

Uncle Moon (Host) (00:03:23)

Right. And once you got into Tsinghua, how did you end up in the Yao Class?

Junchen Jiang (Guest) (00:03:27)

That's an interesting story. I came through math competitions, so the program I initially entered at Tsinghua was the Fundamental Science Class, an undergraduate program combining the math and physics departments. When I arrived, I found there were so many people in that program. Meanwhile, the computer science department had a new class. I didn't even know who Andrew Yao was then. I just knew the computer science department was starting a new class, and you might have to take another exam to get in. So I took it. The funny thing was that the exam tested only math. Only math—and specifically math competition material. Of course, I don't know what the Yao Class entrance exam is like now, but back then it was all math competition questions. Even the interview consisted of math competition problems. Only after I got in did I discover that this wasn't simply a computer science class. It was a theory class.

Junchen Jiang (Guest) (00:03:27)

When Andrew Yao returned to China, he felt China's capabilities in theoretical research were relatively weak. He wanted a class dedicated to theory. That's why the entrance exam consisted entirely of math competition questions.

Uncle Moon (Host) (00:04:27)

So it tested precisely the things you knew.

Junchen Jiang (Guest) (00:04:29)

Right, it played right into my strengths.

Uncle Moon (Host) (00:04:31)

Right. What was the experience like once you were in?

Junchen Jiang (Guest) (00:04:34)

The Yao Class is an interesting place. When it first started, there wasn't much emphasis on admitting people with a particular set of traits. They were simply recruiting the best students. But somehow the exam was all math, so some of the people admitted were exceptionally good at math but didn't have much of a computer science background, like me. Others came in with a specific computing background and were excellent programmers. Everyone in the class was smart, but everyone had different strengths. At the beginning, that actually gave us quite a lot of freedom. With such different interests, it was difficult to insist that everyone develop in one particular direction. Today you have the Yao Class, the Zhi Class, and, I think, AR classes, AI classes, and so on, so the tracks have become much more specialized. Back then, it was more: put the best people together and let them grow on their own. Right? But there was still an emphasis.

Junchen Jiang (Guest) (00:04:34)

The curriculum was very well designed. There was theory and there were other subjects, and it was fairly balanced. But then there were the class's counselors and... At the beginning, the professors who managed the students and supervised their research were all theorists. Naturally, they wanted us to follow the theoretical path. But they hadn't said when recruiting us that only people interested in theory could join. To be honest, I was someone who eventually discovered I wasn't interested in theory. Let me tell you a story about how I realized that. When I first started doing research, I joined a networking group. They handed me a problem. They thought, "You're from the Yao Class, so you must know theory and algorithms. We've got an algorithm problem here. Improve its performance." And I did improve it. I forget by how many times I sped it up. But it was an interesting algorithm. I thought, "This is an algorithmic problem, and I've made a breakthrough."

Junchen Jiang (Guest) (00:05:47)

So I went to a professor in the Yao Class and asked whether it was meaningful. His first reaction was, "You've achieved an improvement. That's good. But can you prove that under certain conditions, this is the maximum improvement you can get?" I said, "I don't care about it. I don't care about that. I'm not interested in its theoretical upper bound. I care about whether I can improve it further." That made me realize something about theory. The focus is more on how deeply you can understand a problem: can you understand it so thoroughly that you can identify things that are impossible? It's about impossibility—some things simply cannot happen. It gives you an upper bound. Proving an upper bound is something theorists particularly like. I don't like that. I like pushing past limits.

Uncle Moon (Host) (00:07:24)

So discovering in the Yao Class that you didn't like theory was one reason you later moved into networking and systems research?

Junchen Jiang (Guest) (00:07:31)

That was part of it, yes. At the time I just felt theory might not suit me. But how I ended up in networking was entirely a matter of chance. I started working on networking in my first or second year. There was a program in the department—or perhaps the school, I don't remember—where each student could get some funding if they found a research advisor. The money would go to the professor to supervise you. A few of my roommates and I said, "Okay, let's find a professor." But whom should we look for? We didn't know anyone. So we listed all the computer science professors alphabetically from the department website. Ha ha. And we called them one by one. I remember the first two didn't answer, or they were away on business. We had no idea what they were doing. The third professor answered: Professor Bin Liu.

Junchen Jiang (Guest) (00:07:31)

He was also traveling, but he very enthusiastically introduced us to one of his undergraduates—one of his PhD students. That PhD student went on to supervise us for three years. His name was Yi Tang. He was an incredibly enthusiastic PhD student. Although he may have been struggling a bit with his own research, and things weren't really coming together, he was a very, very enthusiastic person. From getting us into the lab and finding research topics, to helping us read papers and write papers, he helped us tremendously with all of it. Professor Liu himself was also a very single-minded person, devoted to research. One thing I'm really happy about is that, although I didn't actively set out at Tsinghua to discover which research area I liked, joining that lab at least made me feel that research was really interesting. The professors didn't tell you that you absolutely had to write papers or do particular things. Everyone was quite relaxed.

Junchen Jiang (Guest) (00:08:43)

They were very encouraging when we wrote papers, but they didn't force us to produce more and more of them. That made me very happy. Looking back, being an undergraduate at Tsinghua doing research among a group of PhD students was a really enjoyable experience.

Uncle Moon (Host) (00:09:34)

That's quite different from some other Tsinghua students' experiences. Ha ha. Quite a few Tsinghua graduates I know say their time there was all about competing for the Special Scholarship and competing over grades. But listening to you, it sounds like you were a happy undergraduate in the Yao Class, free to explore and learn whatever you wanted.

Junchen Jiang (Guest) (00:09:56)

Well, I think there's a reason for that. It's not that I lived in a different world from everyone else; my mindset may have been different. I'm someone who likes doing what I enjoy, and I don't want to compete over whatever everyone else cares about. I've always had this view: people in the same circle are all smart, and their interests are generally pretty normal. Right? If you want to spend all four years playing video games, okay, that might really conflict with your goals in life. But for ordinary undergraduates who get into Tsinghua or other good universities, their interests are usually perfectly good ones. As long as you put all your enthusiasm and energy into what you want to do, you may not win the Special Scholarship, you may not have the highest GPA, and you may not come first by the standards everyone uses. But you won't do too badly either. Take me, for example.

Junchen Jiang (Guest) (00:11:08) †

I've always been in the middle of the pack. I was in the middle in the Yao Class, and I was in the middle in my math class at Shanghai High School too. Ha ha.

Uncle Moon (Host) (00:11:14)

But you were in the middle of the very best environment, right?

Junchen Jiang (Guest) (00:11:17)

Right. I admit that what I'm saying may be a little biased. I think that among good students at a similar level, people's interests are fairly comparable. Competing with everyone over the same things they all care about is exhausting. If you want to compete for the Special Scholarship, you need an entry ticket, so to speak. You have to reach a certain level before you can even compete for it. Not everybody can take part in that kind of competition. I'd rather compete with myself over the things I enjoy doing.

Uncle Moon (Host) (00:11:42)

So back then, you didn't take part in those Special Scholarship...

Junchen Jiang (Guest) (00:11:45)

Right, no. I wouldn't even go to the selection sessions for the Special Scholarship. Ha ha. Even sitting in the audience felt like a waste of time to me.

Uncle Moon (Host) (00:11:53)

We have to put that in the show. Ha ha. Junchen thinks being in the audience for Tsinghua's Special Scholarship selection is a waste of time.

Choosing a Research Field and a PhD Advisor

Uncle Moon (Host) (00:11:57)

Ha ha. So after that experience doing research in networked systems, you went on to do a PhD, right?

Junchen Jiang (Guest) (00:12:08)

Right.

Uncle Moon (Host) (00:12:08)

Why didn't you choose AI for your PhD?

Junchen Jiang (Guest) (00:12:11)

That's like asking a PhD student today, "Why didn't you choose computer science—no, computer networking—for your PhD?" Ha ha. Back then AI wasn't even a thing. Ha ha.

Uncle Moon (Host) (00:12:22)

How much of a non-thing was it?

Junchen Jiang (Guest) (00:12:24)

Ha ha. No, you know, I remember this very clearly. When I was doing my PhD—no, in my final undergraduate year—I also spent some time in an NLP group. NLP wasn't particularly hot then. I still remember a PhD student—I don't remember which year he was in—telling everyone at a group meeting, "I've heard about something that's really taking off. This is definitely going to be the future. I really believe in it. It's called deep learning." Ha ha. That was in 2010.

Uncle Moon (Host) (00:12:53)

2010, okay.

Junchen Jiang (Guest) (00:12:54)

Right. That was before AI had taken off. It really hadn't taken off yet. We were still in that phase of computing here in the US after the 2008 financial crisis. Before 2008, it was all networking. Networking was still considered one of the hotter areas in computer science. Of course, everyone could see that computer networking had peaked around the Internet's hottest period, around 2000. Ten years later it wasn't so hot anymore, but nobody could see what the next big thing would be. Nobody saw it. If you remember 2010, that was the mood: everyone was waiting for the next hot thing to emerge. AI didn't immediately emerge then, either. As I remember it, from 2010 to 2015 it was big data and cloud computing. Then, around 2016 or 2017, AI started to take off, with applications such as computer vision coming along.

Junchen Jiang (Guest) (00:12:54)

So if you ask why I didn't choose AI in 2011, I didn't even know why AI would become so popular.

Uncle Moon (Host) (00:14:03) †

Ha ha. I took the college entrance exam later than you, so at the time I would have been in middle school or high school. Ha ha. I had no idea what was happening in the world. Let me think. AlexNet only came out in 2012. But even for quite a while after that, people still weren't really convinced by AI and deep learning.

Junchen Jiang (Guest) (00:14:23)

It wasn't so much that they weren't convinced. People thought this was going to take off—or people working on AI had already recognized deep learning. Like that PhD student I mentioned, who was working with Professor Maosong Sun. He went to a conference and came back telling me, as if it were breaking news, "Deep learning is the future. We have to work on it." And I remember he didn't even know what deep learning was. Back then it was all SVMs, right? Pattern matching and things like that.

Uncle Moon (Host) (00:14:52)

Right, everyone started learning machine learning through things like SVMs and linear regression. Right? Deep learning wasn't really a concept yet.

Junchen Jiang (Guest) (00:14:59)

Right.

Uncle Moon (Host) (00:15:00)

Then you went to CMU.

Junchen Jiang (Guest) (00:15:02)

Right.

Uncle Moon (Host) (00:15:03)

How did you choose Hui Zhang as your advisor, and why did you get on board with networked systems?

Junchen Jiang (Guest) (00:15:07)

First, networked systems... To be honest, I really enjoyed doing networked systems research in that group at Tsinghua, so I wanted to keep doing it. But although I loved the atmosphere in the group, its work was in network security, and I really didn't like security. I felt that security and privacy were also about stopping people from doing things. They prohibit things. They require you to prove rigorously that something is definitely secure. I don't like that kind of rigor. I like breaking through constraints. Ha ha. So when I applied for PhD programs, most of the offers I got were from security professors. They'd look at me and think, "You've worked on security before, so obviously you'll want to do security in the future." None of them realized that I didn't want to do security anymore. CMU's admissions were handled by a committee, so I wasn't assigned to any particular advisor.

Junchen Jiang (Guest) (00:15:07)

When I went to CMU, they had what they call a visit day for incoming PhD students... They admit you as a PhD student, and then you go visit. I spoke with several networking professors at CMU then. But I remember some older students suggesting I talk to Professor Hui Zhang. At the time, he was running a company in Silicon Valley.

Uncle Moon (Host) (00:16:33)

He was already running a company in Silicon Valley in 2011?

Junchen Jiang (Guest) (00:16:34)

Yes. His company had been going since 2004, a long time. Actually, I don't know whether it was 2004 or 2007. In any case, it started in the early 2000s. I didn't know this then; I only learned later that he was getting ready to return—[unclear in the source]—and continue teaching. I contacted him, and he said, "You can come over." He bought me a plane ticket, and I came to California to meet him. What made the deepest impression on me was that, throughout the process, Professor Zhang didn't say much about what he thought I should work on or what he liked working on. He had me stand at the whiteboard in a conference room and explain what I wanted to do. I was very excited: "I want to do this, and this, and that." After listening for quite a while—of course, I imagine he was laughing to himself, thinking, "What is this? This isn't going to work."

Junchen Jiang (Guest) (00:16:34)

But he let me talk because he felt he could only understand what I wanted to do, or what kinds of things I liked, by listening to me. I think he was very perceptive about that. It was something he kept coming back to afterward. Although he had directions he wanted to pursue, when it came to how to work within those directions, he was very good at adapting to each student. After listening to me for so long, even if he didn't like what I wanted to work on, he at least knew what I liked. That's something I... Throughout my time working with Professor Zhang, one thing I deeply admired was that he could see what a person liked doing. Then he would take a problem, shape it around that person's interests, and give it to that student. I think that's a fairly distinctive ability.

Uncle Moon (Host) (00:18:05)

From your experience, we can see that as an undergraduate you worked with a professor who really supported you, and for your PhD you worked with another professor you really believed in. For students who want to do a PhD, is that especially important? Do you have any other advice?

Junchen Jiang (Guest) (00:18:20)

First, those two professors had quite different strengths. Professor Bin Liu, at Tsinghua, gave his students a tremendous amount of freedom. And he was very devoted to research. Talking about research made him happy—the more you talked, the happier he became. Honestly, that wasn't so common among the professors at Tsinghua. At CMU, Professor Hui Zhang was exceptionally good at adapting to each student. To many people he might not have seemed particularly hands-on; after all, he'd reached a certain level. But he was very good at tailoring his approach. He knew what you liked. He had a way of making you feel, as you worked on something, "This is exactly what I love doing." Somehow—and Professor Zhang might not know this—I really loved it. But looking back, I wonder whether he initially got me working on it precisely so that I'd come to like it. Ha ha. Right? I think both approaches are very good for students. One professor gives you his enthusiasm.

Junchen Jiang (Guest) (00:18:20)

The other gets you to generate your own enthusiasm. But Uncle Moon, if I buy two lottery tickets... ...and both of them win, and then you ask me how to pick winning lottery tickets, I really don't know what to say.

Uncle Moon (Host) (00:19:35)

Right. I've done a PhD too, so I know it's not common to keep finding good advisors like that, to be honest. Right? If you're a student who really wants to do a PhD, you should make that a priority and try to find an advisor along those lines. Right?

Junchen Jiang (Guest) (00:19:51)

Right. Right, right.

Uncle Moon (Host) (00:19:52)

While you were at CMU working with Hui Zhang, I noticed from your papers that you also collaborated quite a bit with Ion Stoica, who's now very well known in the industry. I think he was also Professor Zhang's student, so he's your academic senior. What were the biggest things you learned, or experiences you took away, from working with him?

Junchen Jiang (Guest) (00:20:14)

Yes. Ion was one of Professor Zhang's very earliest students, perhaps one of the first one or two. I turned out to be Professor Zhang's last student. After I graduated, I suddenly realized he'd left CMU.

Uncle Moon (Host) (00:20:29)

One of the first disciples, and the last disciple.

Junchen Jiang (Guest) (00:20:31)

Right, right. It was quite a coincidence. Ion was already very senior. When I was doing my PhD, he was a full professor at Berkeley. In the research he did with Professor Zhang and me, he played a role somewhat like a co-advisor. He had many students; I was just one of the students he worked closely with. But he had an ability that I don't think many professors really have. He could have a great many projects, yet maintain a very clear map of how they related to one another. At the same time, he could identify the central issue in each project, and he would keep his eye on that issue every time. I think that's something a professor has to be able to do. It's like being on a big lawn: a student might wander around the grass, exploring here and there. But the professor keeps saying, "There's a tree over there. You need to head toward that tree."

Junchen Jiang (Guest) (00:20:31)

The student may always be looking down at the grass, saying, "I'll go wherever I find something interesting." The student knows the tree is important, but can't lift their head. Ion has a way of getting you to look up every time and see where the tree is. Of course, you could also say that because he has so many projects to manage, he can't remember where each student is. He can only remember where the trees are—where the goals are. But he really is exceptionally good at this. He can keep all the projects and their connections very clear, and tell every student, "This is the direction you need to go. This is your goal. This is your goal."

Uncle Moon (Host) (00:22:14)

Did you eventually learn that ability yourself?

Junchen Jiang (Guest) (00:22:16)

Good question. I haven't yet had the good fortune to supervise that many students. Ha ha. So I don't need that ability—yet. I'm joking. But supervising students is... it's hard to say. Let me put it this way: there's a fundamental difference between being a professor and being a student. As a professor, you can't be the person who solves the problem. If you're the one solving it, you're not acting as the professor anymore. Right? You're the person who creates problems, or explains to others why a problem matters. So I wouldn't necessarily say Ion's ability is unique, but he does this exceptionally well. It's not that other professors don't have it. He's just particularly good at it.

Uncle Moon (Host) (00:22:58)

So a professor needs to play the role they're supposed to play, and within that role Ion has his own particular strengths, right?

Junchen Jiang (Guest) (00:23:04)

Right. He's especially good at that.

Uncle Moon (Host) (00:23:06)

As someone who's worked closely with him, what do you think is the key reason he keeps producing successful open-source projects and companies in every era? We all know that in the big data era there were things like Spark and the Mesos scheduling platform. Now, in the large-model era, there are vLLM, SGLang, and all sorts of powerful open-source systems his students have created, along with the companies built around them. What enables him to keep doing that?

Junchen Jiang (Guest) (00:23:39)

Ion has actually been quite involved with our company and with the open-source project before it. He's also an advisor to our company. My impression of him—of course, I'm not him, so I shouldn't speak carelessly—but after knowing him for so many years, I think he's a person with a vision. He isn't driven simply by whatever his students want to do. Some professors follow their students. Others have a vision of their own. He keeps asking himself, "What's going to matter most over the next ten years, or the next five?" Once he's figured that out, he'll clear away everything he was doing before and put all his energy into what he thinks will matter in the next five years. Not everybody has that kind of resolve. Many professors keep working in an area until there's really no way to publish another paper, and only then change direction. He isn't like that. I don't know whether you remember him working on networking and routing...

Junchen Jiang (Guest) (00:24:50)

...on packet QoS, quality-of-service analysis inside routers. He decided to stop, and that was it. He moved on to peer-to-peer transmission. When research on P2P was at its height, he stopped that too and moved into big data. At the peak of big data, he stopped again and started working on AI serving. He doesn't look at whether an area is currently hot. He asks what will matter most over the next five years, and that's what he works on.

Uncle Moon (Host) (00:25:15) †

Wow, that ability to let go is powerful. In our last episode, the guest kept sharing a life lesson: let go of more things so that you can gain more. I suddenly see a connection here. Another thing you mentioned is his ability to devote himself completely to one thing. I've heard a lot of people say that the strongest ability people like that have is to "keep the main thing the main thing." Ha ha. You really do see shared traits among remarkable people.

Junchen Jiang (Guest) (00:25:43) †

Right.

Video Streaming Research and Industry Needs

Uncle Moon (Host) (00:25:45)

So you spent several years doing your PhD at CMU. Your research area then—I read most of your papers—was, ha ha, networked video... networking.

Junchen Jiang (Guest) (00:25:57)

You know, it's not easy to be this thorough as a host.

Uncle Moon (Host) (00:26:01)

Thank you, thank you. It was video and network analysis, right? Video analysis, or analysis of the networks and streaming transmission involved in video. Could you explain in the simplest, most accessible terms what your research was about and how it relates to people's everyday lives?

Junchen Jiang (Guest) (00:26:18)

Nowadays everyone watches videos on their phones: short videos, long videos, live streams, recordings, films, or TV shows. Before 2010, that was difficult. It wasn't common. When the Beijing Olympics were live-streamed online in 2008, several companies had to come together and build a system specifically for that event. Back then, live streaming and video streaming over the Internet were very immature technologies, and the market wasn't very big either. YouTube was enormously popular when it appeared, but there was really only YouTube. Today, watching video this way feels perfectly natural. The turning point came around 2010. At that time, there were media and online video companies such as Netflix, HBO, BBC, and YouTube... They all suddenly realized, "We have video. We can deliver it straight to users." At least in North America, the number of broadband users had suddenly grown enormously. That gave these companies a very strong incentive to deliver their content over the Internet.

Junchen Jiang (Guest) (00:27:29)

It seems obvious now, but back then it was revolutionary, much like AI today. Every media company needed a video-streaming strategy, just as every company today needs an AI strategy. Right? Every company had to have a streaming strategy. They had to build something like this. But how? It was extremely hard. There had never been a system built specifically for streaming. Previously, everyone had just used web pages, which used the HTTP protocol. HTTP was created for web pages. People hadn't thought HTTP could be used for video streaming. The interesting thing was that when these companies suddenly realized they had to launch video products, they could only use HTTP, because HTTP was the only protocol... ...supported by every network operator. That was a turning point. Everyone suddenly realized, "We have to do this." And HTTP really could work for users with high-bandwidth connections. It worked.

Junchen Jiang (Guest) (00:28:39)

Once that was clear, companies rushed to build their own video products, using this web-page format to deliver video. That happened around 2010, and suddenly everybody realized it was possible. So why did I start working on streaming then? The industry was rapidly developing all these video transmission formats and protocols, but no researchers had thought about how those protocols would affect the network. How could we use principled methods, better abstractions, or better algorithms to optimize the systems that had grown up in this way? When I arrived at CMU to do my PhD, Professor Hui Zhang's company happened to be starting to build that too... ...the streaming system, because they also worked on delivering video over networks. They saw the demand first. Professor Zhang said to me, "Look, nobody in our company can really spend time doing this kind of research. Everybody's an engineer. But we've found this genuinely interesting problem.

Junchen Jiang (Guest) (00:29:50)

It's a new product, new infrastructure, and something everybody has to work on regardless. Right? But nobody has thought about how it ought to be built, or how to make it sustainable, more sustainable. Nobody has thought about what the next generation should look like when it's time to update it." Right?

Uncle Moon (Host) (00:30:26)

From a user's perspective, is the point to make sure that the video playback and streaming services from companies like Netflix, HBO, and BBC feel better to watch?

Junchen Jiang (Guest) (00:30:42) †

Right, and have fewer problems. To get a little more technical, that includes making the video play very smoothly and at very high resolution. People may think that all you need is more bandwidth. But video delivery is actually a very complicated system: from the server to encoding and decoding, the transmission system, the network, and then the web page or embedded player on the user's phone or computer. Every part has to work very well. Video delivery is a huge engineering undertaking. If it isn't done properly, users will be watching videos that keep stuttering.

Uncle Moon (Host) (00:31:18) †

If anything goes wrong at any stage, there's a problem, and the user can feel it. Which of your projects in this area are you happiest with, and which had the greatest impact?

Junchen Jiang (Guest) (00:31:29)

That's interesting. There are two projects that left a particularly strong impression on me. The first was my very first project. At the time, nobody even knew what the problem looked like. Every company had built its own system: Apple had one, Amazon had one, Google had one. Everybody had their own system, but nobody knew what these systems looked like overall, whether they shared any features, or whether they had common problems. Right? In my first year as a PhD student, I remember staying up late for several weeks measuring these video companies' systems. I examined network packets to see what they contained and figure out how each protocol worked. It was reverse engineering: finding out how the protocols actually worked. That experience stayed with me because I was so excited by it. I felt I might be the first person in the world to understand the problems all these companies shared and the challenges they were facing.

Junchen Jiang (Guest) (00:32:40)

That really left an impression on me. The paper turned out well too; it actually won an award. But what stayed with me was the process of being the first person in the world to examine that problem. The second paper was work we did with Ion Stoica, Professor Hui Zhang, and Vyas, my other advisor. What was it about? If video suddenly becomes available to everyone, that's a good thing. Letting everybody watch video is obviously good. But could all that data being delivered to each person serve some other purpose besides letting them watch? Think about what happens every time you watch a video. The video isn't just being delivered to you. Its quality also reflects the quality of the network connection between your computer or phone and the server. Right? If you look at all the video playback around the world together, all those playbacks... ...are measuring the network simultaneously. They're measuring network conditions in every corner of the world. Right?

Junchen Jiang (Guest) (00:33:50)

So we had an idea: with all these video streams and all these video-quality measurements, could we combine them to tell us what network conditions around the world actually look like? Where are there problems? Where has the network slowed down? Where is more bandwidth needed? Right? This is a big data problem. A big data problem.

Uncle Moon (Host) (00:34:24)

It sounds a bit like both big data and the cloud.

Junchen Jiang (Guest) (00:34:26)

Right, right, right. That's also why Ion wanted to work on this project with us while he was working on big data. We were the first to use a big data approach to optimize a network application. Video was just one application. It was a good starting point because there was so much video playback. But the underlying idea was very simple: be the first to use big data to optimize applications at the scale of the Internet. That paper had quite a difficult journey. I remember it was only accepted on the fifth submission. The fifth submission. Before that it was rejection, rejection, rejection. That left an impression on me: doing the first paper on something really carries a lot of... ha ha. If you're the first person to try eating a crab, you can get pinched. Ha ha.

Uncle Moon (Host) (00:35:22)

So when you have too much vision, or work on something too early, you can end up a martyr rather than a pioneer, right?

Junchen Jiang (Guest) (00:35:29)

Let me tell you something interesting. I remember Professor Zhang once giving a talk to new students about how to do research. It wasn't my incoming class; it was after I'd already joined CMU. CMU has an interesting tradition. When new PhD students arrive, each professor—or professors can sign up, I think—gets 15 minutes to talk to them. I remember lots of professors doing it. You can talk about your research, about how to do research, or just ask the students what they want to do. Right? Professor Hui Zhang went up and asked a student to draw three dogs on the board. Why three dogs? He said, "Draw three identical dogs." Imagine three identical dogs on the board—very abstract dogs, just simple sketches. An American student drew them, and then he asked everyone, "What's the difference between these three dogs?" Someone said one of the dogs had a longer tail, that they hadn't been drawn properly and weren't identical.

Junchen Jiang (Guest) (00:35:29)

But then he said, "The difference is that only the first dog is the first dog." For the first dog, finding a bone ahead is a good thing. But even if it finds a pit and falls in, that's a good thing too. It was worth it.

Uncle Moon (Host) (00:36:49)

What about the other two dogs?

Junchen Jiang (Guest) (00:36:51)

Those are the dogs following behind. The follower dogs. Ha ha. They're the followers. His point was that if you want to do the best research, you should be the first dog.

Uncle Moon (Host) (00:37:03)

But if the first dog falls into a pit, the other two see it fall in and go around. Isn't that pretty nice for them?

Junchen Jiang (Guest) (00:37:09)

That's what I'm saying: if you want to do the best research, you have to take that risk.

Uncle Moon (Host) (00:37:14) †

Right, there's a chance you'll fall into a pit.

Junchen Jiang (Guest) (00:37:15)

Right, but there's also a chance you'll get the bone first. I think good research requires taking a little risk. To be honest, in computer science research it's actually very hard to fall into a pit you can't climb out of. Ha ha. As long as you're the first to work on something, you can always produce something good.

Uncle Moon (Host) (00:37:39)

That seems to have quite a profound meaning. I wonder what our viewers will make of it. Okay. Ha ha, right.

From Content Delivery Networks to an AI Data Layer

Uncle Moon (Host) (00:37:46) †

Let's come back to your research. Later you moved into research on infrastructure for large models. But when you were working on networks and video streaming, there were also lots of new concepts, such as CDNs.

Junchen Jiang (Guest) (00:38:00) †

Right.

Uncle Moon (Host) (00:38:00) †

Right?

Junchen Jiang (Guest) (00:38:00) †

Right.

Uncle Moon (Host) (00:38:01)

Content delivery networks.

Junchen Jiang (Guest) (00:38:02)

CDNs, or content delivery networks, already existed in the web era, the earlier age of web pages. They were created for web pages. But when video delivery took off, CDNs had another boom.

Uncle Moon (Host) (00:38:17)

Right. It felt incredibly popular then, when I was reading the papers from that period.

Junchen Jiang (Guest) (00:38:20)

Right, right, right.

Uncle Moon (Host) (00:38:21)

Do you think that was mainly because the network was the bottleneck?

Junchen Jiang (Guest) (00:38:25)

Let me put it this way. A CDN, a content delivery network... You're right that its importance for video streaming lies in addressing the network bottleneck in the middle of the delivery process. It puts the video closer to the user, which removes that intermediate bottleneck. But the most important thing, looking back, is why CDNs mattered in the web era, mattered in the streaming era, and still have a presence now. It's because whenever an application reaches a certain scale—when a certain proportion of humanity is using it, frequently enough and in large enough numbers—it inevitably needs a data layer. A layer that makes the distribution of data... ...its sharing and analysis more efficient. You will always run into this problem. In the web era, it was the CDN. In the video era, CDNs changed their approach and solved the problem in a new way. In the big data era, it was Spark and the Hadoop data file system—the data layer of that era.

Junchen Jiang (Guest) (00:39:36)

Every era has something like this. The AI era will have it too.

Uncle Moon (Host) (00:40:08)

Right. That's why I wanted to ask about this. In the AI era, the most distinctive kind of data at the moment seems to be KV cache. Once the Transformer architecture took off and GPT proved itself, everyone suddenly discovered, "Why is KV cache so huge?" Right? So is your decision to work on KV cache connected to your earlier background and to anticipating what would happen next from those previous cycles?

Junchen Jiang (Guest) (00:40:33)

The connection is enormous.

Uncle Moon (Host) (00:40:36)

That's exactly the connection I want to hear about.

Junchen Jiang (Guest) (00:40:37)

First, we did make a prediction. But it's still a prediction. We can put it online and see whether it turns out to be right. Right? What is that prediction? KV cache—or internal memory like KV cache that models generate themselves—will become a more important type of data than data humans can understand.

Uncle Moon (Host) (00:41:04)

By memory, you mean KV cache?

Junchen Jiang (Guest) (00:41:06)

Right, KV cache. KV cache is a Transformer's memory. The Transformer architecture is the most popular and most widely used architecture right now. It may well remain the most widely used in the future, because the people who know Transformers may continue to know only Transformers. Industry has inertia. An architecture that's popular now is quite likely to stay popular. If Transformers remain popular, then KV cache is the model's own memory. Our prediction is that this memory, the model's memory, will become the most important kind of data. Why? Because everyone—or a sufficiently large proportion of humanity—is interacting with these Transformers very frequently. The internal memories those Transformers generate will then be the most important data.

Uncle Moon (Host) (00:41:53)

Back in your earlier era, people put videos in different data centers and different CDN networks so everyone could watch them. In this era, perhaps KV cache will develop the way you're envisioning. Right? Every person generates a huge amount of KV cache through their interactions with AI.

Junchen Jiang (Guest) (00:42:13)

Right.

Uncle Moon (Host) (00:42:14)

Then, whenever they continue interacting with that AI, perhaps they also need a CDN-like data layer to help them retrieve this more quickly and efficiently...

Junchen Jiang (Guest) (00:42:26)

You know, what you just said gives me an analogy. You could think of KV cache as video for large models to watch. Why do people watch videos?

Uncle Moon (Host) (00:42:41)

Because they're exciting and stimulating.

Junchen Jiang (Guest) (00:42:42)

Right, exciting and stimulating. But think about reading a novel versus watching a film adapted from that novel. Which do you find more exciting and stimulating?

Uncle Moon (Host) (00:42:53)

Definitely the film.

Junchen Jiang (Guest) (00:42:53)

The film, right? And which is more efficient to get through?

Uncle Moon (Host) (00:42:57) †

Watching the film. The book... ha ha.

Junchen Jiang (Guest) (00:43:00)

Well, there's a balance. If you go through it too quickly, it's no longer exciting or stimulating, right? But if you compare reading the book carefully with watching the film carefully, the film is definitely more efficient.

Uncle Moon (Host) (00:43:10)

More... you can finish it in two hours.

Junchen Jiang (Guest) (00:43:11)

Right. Why does a film give people that stimulation more efficiently? Because film is a form people can take in more readily. It turns something linear, text, into something three-dimensional, presented right before your eyes, something you can understand directly. KV cache is a bit like video for large models to watch. You can think of it that way. Of course, the analogy isn't exact. After all, KV cache is generated by the model itself, whereas video isn't generated by the person watching it, or by their brain. Right? But when a model sees KV cache, it can immediately know what's in it. It can immediately understand the meaning represented by that cache, because it's something the model generated itself. Also, video and KV cache are both three-dimensional tensors, right? 3D tensors. Ha ha. So in many, many respects, it's like video for large models to watch. Think of it as video that only a large model can understand. Humans can't read KV cache.

Uncle Moon (Host) (00:44:15)

So you're managing and modifying large models' memories. We can explore that in more detail later when we talk about LMCache and the startup project.

After the PhD: New Directions and Industry Experience

Uncle Moon (Host) (00:44:25)

Let's continue with what happened next in your career. After your PhD, I believe you first went to Microsoft Research and then became a professor at the University of Chicago. What was your thinking behind that?

Junchen Jiang (Guest) (00:44:40)

That's a fairly standard path. In North America, after finishing a PhD and before becoming a professor, people typically take a gap year. You need a year to rest—well, it isn't really rest; you're working during that year. But honestly, without that year you're exhausted. You've just finished a PhD, and people who land faculty jobs have generally worked themselves mercilessly during it. If you immediately become a professor, the tenure clock starts ticking right away. Your tenure review comes up in five years, so you have another deadline. Right? Taking a year in between is a pretty common practice. Lots of people do it. As for why I went to Microsoft, I wanted to use that year to find a new direction, ideally something not too far from what I'd done before. During my PhD, working on big data was, in a sense, AI for networking: applying AI to networked systems or network applications, such as video delivery. Right?

Junchen Jiang (Guest) (00:44:40)

But I'd always thought that building systems for AI... ...was also very interesting. But I never wanted to work on training systems. Very early on, I felt that training systems would eventually cease to be the central issue. You train a model once and then use it for a very, very long time. Most of the time isn't spent on training. Most of a model's life cycle is spent on inference, on being used by people. Once the range of applications grows, inference will always be the bottleneck. At that time, the really popular inference application was video analytics, or computer vision. Computer vision was the only area with large-scale inference applications then. They look small-scale now, but they were large-scale at the time. I went to Microsoft because there was a group there that wanted to start building video analytics systems. In fact, that's also what I worked on in my first few years after moving to the University of Chicago.

Uncle Moon (Host) (00:46:58)

I saw that you had at least three stints in industry during your PhD. How did those experiences help you with your PhD, finding a faculty position, or doing research?

Junchen Jiang (Guest) (00:47:13) †

Well, um, to be honest, what I'm about to say may come as a bit of a shock to quite a few students. Ha ha. I did three... Before becoming a professor, I had four experiences in industry. Three were at Microsoft Research, and one was at a company doing engineering and building a product. That product-building experience left the strongest impression on me, and it helped my research the most. The three stints at Microsoft Research all resulted in papers—or at least two definitely produced papers that were accepted. But I think the time I spent in industry directly writing product code and doing engineering work helped my research the most. Why? My first internship was also at Professor Hui Zhang's company. At the time they were thinking about building a video player. A video player is a very, very engineering-heavy, low-level thing to build. In fact, even engineers don't want to do it.

Uncle Moon (Host) (00:48:23) †

Right. As a researcher, I hardly ever saw papers about video players.

Junchen Jiang (Guest) (00:48:27)

Exactly. Nobody knew how you were supposed to write a video player. I remember the player I worked on at Conviva was an Adobe player. Nobody even knew the programming language—they had their own language for it. Then, later, I also worked on the Apple one, that whole system written in C. When Professor Zhang sent me to do that internship, he told me, "Don't think about research. Don't think about research at all during this internship. Just work on the product." Right? But looking back, that was the internship I learned the most from. Why?

Junchen Jiang (Guest) (00:48:27)

Because afterward, among all the PhD students and professors working on networked video delivery, I was the only one who knew what actually happened inside a video player: what kinds of optimizations a player could support, which ones it couldn't, which optimizations could actually help, which couldn't possibly help, and what engineering problems it had— problems that couldn't be solved yet but might be solvable in a couple of years, and problems shared by different video players from different vendors. You only see these things by really getting into industry, immersing yourself in that environment. So I often tell my students: if you go into industry for an internship, or just to gain some experience, go without that research mindset. You have no idea what problems or interesting things you might encounter. Don't take your prejudices as a researcher with you. Really go in thinking, "I'm here to be an engineer. For three months, I'm handing myself over to this job."

Junchen Jiang (Guest) (00:49:37)

But afterward, you can look back as a researcher. You've left after three months, and you can ask: what did I learn? Beyond writing code, I now understand what actually happens in industry. That's an enormously valuable asset for your future research. Of course, I don't encourage students to enter an industry that has absolutely nothing to do with their research— but if it is related, go. And go without that researcher bias that you have to do research or write a paper. Just go and do the engineering.

Uncle Moon (Host) (00:51:01)

With a view like that, Elon Musk would love you! Hahaha. He really dislikes the term "researcher."

Junchen Jiang (Guest) (00:51:06)

I'm not saying students should become full-time engineers! Hahaha.

Uncle Moon (Host) (00:51:10)

Right, but it's a fascinating perspective. After hearing your explanation, I think many listeners will take something different away from it. My immediate reaction was that a good researcher first needs to become a good engineer and immerse themselves in industry to experience what's happening there.

Junchen Jiang (Guest) (00:51:29)

You're right, but also not quite. What I mean is that you go into industry to see what people are actually doing. But to see what's really happening inside, you have to become one of them. You have to get in among them. Only then can you see what's going on. You need to hear what those engineers are arguing about every day to understand what's happening, what's difficult, and what's easy. Academia can very easily turn into a bubble. You say a problem is important, I say it's important, someone else says it's important, and then I decide it must be important. But you may never have seen what the problem actually looks like. Right? Or you just read about it in other people's papers. All those papers work on that problem, so you do too. Then you'll always be following other people. What can you see in industry? The problems that engineers on the front lines actually face.

Junchen Jiang (Guest) (00:51:29)

That doesn't mean all those problems are necessarily important— but if there are important or particularly interesting problems, those frontline engineers aren't going to write papers about them. They don't care about that at all. You have to be there to see those problems. Once you've seen them and your summer internship is over, go back and think: could this become a paper? Often, by the time you've spent a year writing that paper, the problem really has become important. A year later, it may no longer be something only frontline engineers encounter. A great many engineers may be facing it.

Uncle Moon (Host) (00:53:13)

Right, but after hearing all that, I don't feel you're disagreeing with me. You're supporting what I said: in industry, a researcher needs to adopt an engineer's mindset.

Junchen Jiang (Guest) (00:53:21)

Yes, an engineer's mindset. I'm not saying they have to become a good engineer.

Uncle Moon (Host) (00:53:24)

Right. Then use that engineer's mindset to absorb as much as possible about the real problems occurring in industry.

Junchen Jiang (Guest) (00:53:29)

Exactly.

Uncle Moon (Host) (00:53:29)

The specific problem you're solving now might, in two or three years—or even just one year—be something every engineer at other big companies is working on.

Junchen Jiang (Guest) (00:53:37)

I may have misunderstood you at first. I thought you meant becoming a good engineer, really building up your engineering abilities. I don't think you can do that in three years—three months, a summer, a day. But you can join them with the right mindset. Right? Do your best to become part of the group. That's how you see the real problems.

Uncle Moon (Host) (00:53:56) †

Right. Some students, and even some professors, might think, "I don't want to participate in real projects in industry." But that can leave your research like a castle in the air.

Junchen Jiang (Guest) (00:54:06) †

Yes, staying in that bubble.

Uncle Moon (Host) (00:54:08)

Right, the proverbial spherical chicken in a vacuum. That's probably the way to put it.

Junchen Jiang (Guest) (00:54:10)

Exactly, exactly.

Uncle Moon (Host) (00:54:12)

Personally, I think that makes a great deal of sense. Computer science should be an engineering science, shouldn't it? You still need to engage with the real world through engineering. Right?

Junchen Jiang (Guest) (00:54:27)

Yes.

Uncle Moon (Host) (00:54:28)

And then abstract problems from that experience.

Junchen Jiang (Guest) (00:54:29) †

Yes.

Uncle Moon (Host) (00:54:29) †

Right?

Junchen Jiang (Guest) (00:54:30) †

Exactly. Otherwise, you might as well work on pure theory. Right? Hahaha.

Becoming a Professor, Asking Questions, and Learning from History

Uncle Moon (Host) (00:54:35)

That brings us to your faculty career. Moving from being a PhD student to a professor, what was the biggest difference, and what was the greatest difficulty you encountered?

Junchen Jiang (Guest) (00:54:45)

The biggest difference is that being a professor—a PhD advisor—is very different from being a student. As a student, you're the person who solves problems. As an advisor, you're the person who poses problems or guides other people toward posing them. You can no longer be the problem-solver. That's something many excellent PhD students struggle with when they become advisors. They think, "I'm extremely good at solving problems. I'm an expert problem-solver." But once you're a professor, you can't be the one solving the problems. You have to restrain that enthusiasm. You can't solve the student's problem for them. If you both pose the problem and solve it, what is the student doing? There's no point in that. Right? You have to become someone who poses problems or helps others learn to pose them. Honestly, that's difficult. Even excellent PhD students don't learn this during their PhD training.

Junchen Jiang (Guest) (00:54:45)

It's rather like an engineer being promoted into management— and then finding they can't manage people, because they never had that skill before. I'm not saying PhD students can't learn it when they become advisors. But it is a new skill they have to acquire. Sometimes, when an excellent PhD student tells me they want to become a professor, my first question is: do you prefer solving problems or posing them? If you prefer solving problems, think carefully. Think twice. You may not be suited to being a professor.

Uncle Moon (Host) (00:56:26)

So you have to respect the role you're in. But that also raises another issue: you need to judge which problems have real value, including whether the problems your students propose are valuable.

Junchen Jiang (Guest) (00:56:38)

Let me put it this way: whether a problem is valuable really depends on how you look at it. In North America—or at excellent universities such as Tsinghua and Peking University—professors at the best schools have a luxury, an advantage other people don't have. You really have an opportunity to change people's minds. You can take something nobody considers important and make it important. You can change the course of an industry. Many professors think their goal is to train students, write more papers, win more grants, and have their students spread out and establish themselves, leaving a substantial academic legacy. But honestly, if you teach at one of the best universities, you have that freedom. Why are you given it? So you can change people's minds. You asked me when a problem is good or bad. Sometimes a student proposes a problem and my first reaction is, "No, that won't work." But then I think again: perhaps it really could work.

Junchen Jiang (Guest) (00:57:49)

Perhaps if the student and I really pursued that direction, we could change the world. We could change how people think about that kind of problem.

Uncle Moon (Host) (00:58:00)

It seems we're still not actually disagreeing. I'm asking whether the problem is important. You're saying it might not be important at the time, but could become very important in the future.

Junchen Jiang (Guest) (00:58:10)

Exactly, exactly.

Uncle Moon (Host) (00:58:10)

Do you see? We end up in the same place by different routes: finding problems that may become important next and working to solve them.

Junchen Jiang (Guest) (00:58:19)

Exactly. But which problems really will become important next? That's the million-dollar question. There's no certainty, and everyone has a different view. Some professors—and I probably belong to this group—look at recurring patterns in history. We think something is important because something similar happened before and was very important. For that, you need some historical experience. You need to know what mattered in the past. If you've done a PhD, think about those so-called advanced courses: advanced algorithms, advanced operating systems, advanced networking, advanced databases. What are those courses? They have you read classic papers. Think back to when you were doing your PhD: you read those classics. Reading classic papers isn't really about learning to write a classic paper. Much of what's in those papers is already dated. Reading them tells you what mattered in history. You're actually taking a history class. Often, people think— you don't really need to take those PhD courses.

Junchen Jiang (Guest) (00:59:30)

But honestly, I tell my students that they're the very best courses. They genuinely teach you the history of our field, or another field. Only when you've seen the important things that happened in the past can you start recognizing what might happen in the future. Of course, that also depends on how well the professor teaches. If the professor simply makes students present the papers themselves, that purpose is lost. I've taught advanced courses a few times, and I always prepared the PhD-level material myself. I told the students what had happened historically and encouraged them to think about what might happen next. It's natural: you need to learn history to understand the future. That's one way to identify important problems at a broad level. Another is to work very closely with industry, as I mentioned. As industry moves forward, you're the first person to see that gradient, that delta. Then you go and work on it. That's another approach.

Junchen Jiang (Guest) (00:59:30)

As for other approaches, I haven't come up with a better one yet. Haha.

Uncle Moon (Host) (01:00:35)

Hearing you explain it that way, I think studying the history of science is important not just for PhD students, but for everyone—even people who don't do research.

Junchen Jiang (Guest) (01:00:43)

Yes, extremely important. Of course, the history of science matters especially for scientists, just as art history matters for artists, and literary history matters for people studying literature. History is enormously important. I always tell my students: even the history of Silicon Valley is worth reading. Look at PARC—I don't know whether you know it—the Palo Alto research center, that laboratory in Silicon Valley that first developed Ethernet and the graphical user interface. Or Bell Labs. These famous events and laboratories in computing history, along with the history of your own field, are an incredibly valuable resource. Usually, people don't teach them. They think they're just old relics with no use today. Do you know what students usually think when they read an old paper? "Oh, this idea was proposed twenty years ago, and I'm still working on it now." That's often their reaction.

Junchen Jiang (Guest) (01:00:43)

But these genuinely advanced courses—I don't know how to translate 'advanced' into Chinese— advanced networking, advanced systems, advanced operating systems, advanced databases, advanced AI, advanced algorithms: very often, these courses are teaching history. People haven't really understood why they're reading those papers.

Uncle Moon (Host) (01:02:09)

That's a genuinely new perspective: treating all those advanced computer science courses as history courses.

Junchen Jiang (Guest) (01:02:16)

They are history courses.

Uncle Moon (Host) (01:02:17)

Wow. I honestly hadn't thought about it that way before. But I do enjoy reading things like On Top of Tides, or books about the rise and fall of Silicon Valley companies. Even watching television dramas! Hahaha. They give you some idea of what happened in Silicon Valley and in the history of science.

Junchen Jiang (Guest) (01:02:34)

Yes, exactly.

Uncle Moon (Host) (01:02:34)

When you really think about it, those things are quite helpful. Later, you became a professor leading students further into video analytics, and now you've started working with them on LLM systems. Throughout that journey, given that you yourself came from Tsinghua's Yao Class, it wouldn't be realistic to recruit only Yao Class graduates.

Junchen Jiang (Guest) (01:02:59) †

No, that would be impossible.

Uncle Moon (Host) (01:03:00) †

Impossible, right. So how do you approach working with and mentoring your students?

Junchen Jiang (Guest) (01:03:06)

When people talk about recruiting students, the first thing they think about is the applicant pool. I do think that matters, but it isn't everything. I've probably been lucky: I've recruited quite a few students from prestigious universities who ultimately did very well, exceeding my expectations. I think a large part of why they initially joined me was that they liked my research direction. But as time went on, I realized that these students all had distinctive personalities. Very distinctive. Some were strong engineers, some were strong thinkers, some were good at writing papers, some were good at working incredibly hard, and some were strong communicators. Everyone had different characteristics. I find that fascinating. Every time I meet with a student, I'm thinking: how can I persuade this person to do it this way? How should I explain that they should try another way?

Junchen Jiang (Guest) (01:03:06)

Sometimes I even think: this student never listens to me— if I want them to go east, should I first tell them to go west, so they'll naturally head east? Hahaha. Sometimes an advisor really does have to put a little thought into that. But the most important criterion in recruiting isn't a student's background. It's whether they have a particular quality you're looking for. For me, being from the Yao Class isn't the most important feature. The people whose thinking is closest to mine are students who have done academic competitions. My two co-founders, Kuntai and Yihua, both did competitions. Neither did math competitions, so they weren't the same competitions I did: one did chemistry, the other computer science. But competition students share a quality that really resonates with me. What is it? Their thinking isn't linear, and they're proud of that. They were both admitted to university through competition-based recommendations.

Junchen Jiang (Guest) (01:04:17)

One prerequisite for that route is that you have to— accept the risk of not having prepared thoroughly for the college entrance exam. If the competition doesn't work out, you really do have to sit the exam. I'm not saying they couldn't do well in it—I believe they could certainly get into Peking University. But they take pride in that ability to make leaps in their thinking, to do things other people haven't thought of. That ability is very important in academia. Some professors like recruiting obedient students who are willing to work hard and produce results. I may not be the right advisor for that kind of student. Hahaha.

Uncle Moon (Host) (01:06:09)

That's not a low bar either! It doesn't sound any easier than recruiting a student from the Yao Class. Hahaha. Right?

ChatGPT's Internet Moment

Uncle Moon (Host) (01:06:17)

All right. Later, you moved from video to large language models. How did that transition happen?

Junchen Jiang (Guest) (01:06:23)

That's a very interesting story. We'd never thought about it. We'd never worked on LLMs before. We had always done video and networking, and at most some AI systems work, but we'd never touched LLMs. I first got into them in early 2023, a month or two after ChatGPT came out. It was everywhere online. You'd go to a restaurant and some random person at the next table would be talking about ChatGPT. That's how popular it was after just two months. At that point, almost every student was asking their advisor, "Can I work on large language models? Can I do something with ChatGPT or GPT? It's so hot right now." My own students asked me too. I remember most professors' first reaction was: "This is a buzzword. It may disappear in a couple of days. Besides, it's just an application. Are you really going to change your systems research direction because of one company's extremely popular application?" "Of course not. That's hype.

Junchen Jiang (Guest) (01:07:35)

It's hype, not a real trend." That was the reaction many of my colleagues had when their students asked them. Honestly, it was my reaction too. But my students kept asking. I remember there was a SIGCOMM deadline in January 2023, and they almost didn't even want to work toward it anymore. They kept asking me about this instead. So I said, "All right, let's talk. Let me think about whether there's really something worth doing here." Once we talked, it turned out to be a very big deal. We thought this might genuinely be an inflection point. The difference wasn't how good the application itself was, or how different the underlying AI was. Everyone knew that ChatGPT was a Transformer, and Transformers had been around for years. Why suddenly work on Transformers now? Because AI—or ChatGPT—was the only, or at the time the first, AI product with the potential for everyone to use it at any moment— all the time.

Junchen Jiang (Guest) (01:08:47)

No previous AI product or technology had reached that level. Before, you trained a particular model for a particular scenario to do particular things. ChatGPT was the first time this happened. Think about our field, networked systems: what was its most important turning point? The early 2000s, or the late 1990s, when people first realized that all computers could be interconnected, that applications could connect everyone, and that everyone could want to be online all the time. Right? We felt ChatGPT was the first time AI had that kind of Internet moment.

Uncle Moon (Host) (01:09:39)

You reached that conclusion after just a few discussions with your students?

Junchen Jiang (Guest) (01:09:43)

We really did. After a few conversations with them, I thought: this really might be it. Sometimes something just clicks. Once you realize how different an application is, your mind keeps asking: is it really different? Is it genuinely different? And every time you think about it, you see that it really does resemble something that happened in history.

Uncle Moon (Host) (01:10:04)

But back then, it was just a chatbot!

Junchen Jiang (Guest) (01:10:07)

Exactly.

Uncle Moon (Host) (01:10:07)

It was so far behind the coding agents and all the other agents we have now.

Junchen Jiang (Guest) (01:10:12)

But think about it. Maybe they chose the right application. Maybe the chatbot was simply a good choice. But a chatbot is something everyone needs, isn't it? I remember going out to eat and hearing people at the next table—maybe insurance salespeople—talking about ChatGPT. I realized this was the first time AI had really broken out beyond its own circle. It wasn't just specialists who wanted it. Everyone wanted to use it. That's the Internet moment.

Uncle Moon (Host) (01:10:42)

Wow. We were actually running a startup back then, but we thought this was just a huge wave of hype. Hahaha. Looking back now, including at people I've interviewed recently, whether you got involved before then or in January 2023, you were still an early adopter.

Junchen Jiang (Guest) (01:11:01)

Yes.

Uncle Moon (Host) (01:11:02)

You could benefit from the opportunities that unfolded afterward.

Junchen Jiang (Guest) (01:11:05)

I really need to give the credit to my students, including Yuhan and Kuntai. They were among the first people pushing me to look at this.

Uncle Moon (Host) (01:11:16)

Hearing that makes me feel old just two or three years after graduation. You lose that student instinct to rush toward anything new. As a child, you'd see a Subor learning computer and want to play with it, however rudimentary it was. But when I saw ChatGPT, I thought: this thing can have a conversation, but how does it help my productivity or daily work? I also tried GitHub Copilot at the time, and not a single line of code it wrote was usable. It was just very, very difficult to use.

Junchen Jiang (Guest) (01:11:43)

People were looking at the quality. We were looking at how broadly it could be used, and how many people were interested.

Uncle Moon (Host) (01:11:49) †

Right. Later, I realized that earlier applications, from AI video analytics to audio analysis, had been very fragmented. Each model handled one thing. It wasn't widespread. Each—

Junchen Jiang (Guest) (01:12:05) †

Right.

Uncle Moon (Host) (01:12:05) †

product's AI feature was just—

Junchen Jiang (Guest) (01:12:07)

Right.

Uncle Moon (Host) (01:12:09)

one part of it, neither especially large nor insignificant. But after that ChatGPT moment, it seemed as though AI itself had become the product.

Junchen Jiang (Guest) (01:12:13)

Yes.

Uncle Moon (Host) (01:12:15)

A product anyone could use.

Junchen Jiang (Guest) (01:12:15)

Yes. It wasn't necessarily something everyone could use immediately, but everyone wanted to use it, or thought it could be something they'd want. It's like the Apple computer when it first came out. Steve Jobs was overmarketing it: he said you could do all these cool things, but you couldn't actually do that many cool things yet. People still wanted to buy it because he'd captured their imagination. Once something captures people's imagination, it will spread. That cycle, that flywheel, has started turning. Of course, it was OpenAI—or Google at the beginning—that really got it moving. But once you realize a positive feedback loop has begun and everyone wants this thing, people building infrastructure should recognize it as the next Internet, the next big data, the next cloud, the next AI.

Uncle Moon (Host) (01:13:11) †

That's true. If you work on infrastructure, you need to identify something that will operate at sufficient scale. That's where optimization can have the greatest impact.

Junchen Jiang (Guest) (01:13:22) †

Yes.

Uncle Moon (Host) (01:13:22) †

Optimizing infrastructure around minor details—hahaha, you know what I mean—the impact you can make is negligible, you might say.

Junchen Jiang (Guest) (01:13:30)

Yes, yes.

Uncle Moon (Host) (01:13:31) †

So infrastructure people need to focus on the big trends.

Junchen Jiang (Guest) (01:13:33) †

Exactly.

Uncle Moon (Host) (01:13:33) †

And make all kinds of underlying optimizations to support those trends.

Junchen Jiang (Guest) (01:13:36)

To see the big trends, you have to study history.

Uncle Moon (Host) (01:13:39)

We're back to that earlier lesson—a callback! So infrastructure people really should read more history of science.

Junchen Jiang (Guest) (01:13:46)

Haha. At least the history of computing technology.

Choosing KV Cache and Open-Sourcing LMCache

Uncle Moon (Host) (01:13:50)

How did you then choose KV cache as your research topic? From that point on, I saw your research direction shift, with everyone in the group working on KV cache.

Junchen Jiang (Guest) (01:14:01)

Some students still needed to finish their papers so they could graduate. Right? But for the students who wanted to work on KV cache, or inference optimization for large language models, I told them, "Go ahead. It's fine. Go for it. At worst, we don't publish for a year." I was still two years away from my tenure review then. I said, "Don't worry. Finish your previous papers, and then work on something new. I don't care whether you publish new papers. Even if I don't get tenure, that's fine, because this is genuinely important." I told them that if they had good suggestions or ideas for LLM inference, we'd discuss them, because to me this looked like the next Internet. Why KV cache specifically? That was a very practical consideration. We came from networking. Networking needs large amounts of data. If there isn't much data, there isn't much of a networking challenge.

Junchen Jiang (Guest) (01:14:01)

So we thought: input tokens, all this text you're feeding in— what's so large about that? Right? First we looked at whether there was video input. But at the time, multimodal capabilities at OpenAI and in LLMs generally weren't very good. You couldn't really work with video, and generating video was extremely slow. So if we couldn't do video, what could we do? Later, I realized that the KV cache inside the model was a three-dimensional tensor, just like video—a 3D tensor. And you could reuse it. You could store it, reuse it, and transfer it between models. Two models could share the KV cache, share that internally stored memory. Now that was something interesting. It was right in our wheelhouse.

Uncle Moon (Host) (01:15:59)

Then you built projects like CacheGen, CacheBlend, and LMCache. You even open-sourced LMCache.

Junchen Jiang (Guest) (01:16:08)

As I said, open source wasn't the original plan. Initially, we wanted to build something we called the Internet of Agents. We were discussing it with several database professors. We imagined something like Sky Lab—no, not Sky Lab—something like AMPLab: bring together people working on databases, networking, and AI, and establish a research direction around optimizing LLM inference. We would work on KV cache; others might work on KV cache, context, databases, and so on. What did we do first? KV cache compression, because KV caches are so large that you need compression. Then we worked on CacheBlend, non-prefix reuse: making it possible to reuse KV caches outside the prefix. Once we'd done that work, I started giving talks. I'd say to people in industry, "Look, reusing KV caches offers enormous benefits, so you must already be storing them. You must have a KV cache storage system, right? In that storage system—" "if you're storing so many KV caches, you'll need compression, right?

Junchen Jiang (Guest) (01:17:15)

Each one is huge. And if you want more reuse, you'll need blending. Look, we've already written the research papers. Where's your KV cache storage system? We'll help you deploy these optimizations." And they'd say, "What's a KV cache storage system? We don't have one." Hahaha. They'd say, "I know KV caches can be reused, but right now they only exist in GPU memory. These inference engines discard the KV cache once a request finishes generating. We've never stored KV caches. What is this storage system you're talking about?" That's when I realized we'd got too far ahead. Hahaha. Of course, we had two options. We could give up and do something else: industry wasn't ready, and people couldn't use what we gave them. Or we could keep working and wait until they wanted it. But we couldn't wait. I thought, "Since you haven't built it, we'll build this technology for you.

Junchen Jiang (Guest) (01:17:15)

We'll open-source a project that lets you store KV caches in a system, and then you can use our optimizations." So we worked on KV cache to let industry use our research systems—no, our research ideas and papers. What happened was that once the open-source project succeeded and more and more people used it, we went on to start a company. But as the project became popular, people started saying, "Junchen, aren't you the people who build LMCache, the KV cache storage system? Who does the compression work, or that non-prefix compression—non-prefix reuse research? Do you work on that?" I'd say, "That's what we were doing in the first place!" Hahaha. You couldn't use those ideas, so I built an open-source system for you. Open source wasn't the original plan. But after we'd spent more time building the system, people began assuming that had been our aim all along.

Uncle Moon (Host) (01:19:21)

Listening to that story, doesn't it take you back to the beginning of your PhD? You saw video websites growing wildly in industry, and you abstracted the problems. Now, in the LLM era, you've developed KV cache work, compression, and prefix and non-prefix reuse. Then you look back at industry and realize it's not ready. It's still growing haphazardly, running a bit of inference here and there with different open-source models. The overall system is still fragmented and developing chaotically. People haven't systematically taken it further, toward something more economical and efficient.

Junchen Jiang (Guest) (01:19:52)

Honestly, doing this takes a great deal of courage. Looking back, we may have been a little oblivious to what we were taking on. Building an open-source project, creating an open-source ecosystem, doing something industry might have done on its own—that really takes courage. You're effectively using an academic research group to challenge the whole industry. Industry would build this system sooner or later. By the time you finished, companies might simply copy it, or decide they could build something better. But we may have bet correctly. We really may have. By the time industry actually wanted to build the system, it was already very late. LMCache already had a fairly complete feature set. So many companies now simply reuse LMCache; some even resell it. Others work with us commercially on LMCache.

Junchen Jiang (Guest) (01:21:03)

But looking back two or three years, when we were just starting LMCache, if I could go back and ask my earlier self whether we should build an open-source project, I'd say it was an extremely risky thing to do. Perhaps we were only looking at success stories. Spark was one. Spark effectively emerged by stepping over industry competitors. So many companies wanted to build similar projects, and Spark came out on top of their corpses. It forcibly knocked out those industry projects one by one. LMCache has done something similar. We simply didn't realize at the time how great a challenge it was.

Uncle Moon (Host) (01:21:47)

Then, at this year's GTC, we saw Jensen put the LMCache logo up on the big screen. What did that moment feel like?

Junchen Jiang (Guest) (01:21:56) †

At that moment, I pulled out my phone and started taking pictures as fast as I could. I remember the camera was a little blurry, so I frantically wiped it with my clothes. Wipe, wipe, wipe! Hahaha.

Uncle Moon (Host) (01:22:07) †

All right, we'll put that video—not the video, the photo—up on screen in a moment. Okay?

Junchen Jiang (Guest) (01:22:13)

I don't know whether I still have those first blurry pictures I took. Eventually, I thought: forget taking photos. There was a livestream online, so I just took screenshots of that.

Uncle Moon (Host) (01:22:23)

Makes sense.

From Open Source to Starting a Company

Uncle Moon (Host) (01:22:24)

What made you feel you had to start a company with your students, even moving your family to Silicon Valley?

Junchen Jiang (Guest) (01:22:31)

Those are two different questions. Starting a company with my students and moving to Silicon Valley were separate decisions. Honestly, starting the company was a choice we almost had to make. The students working on the project were about to graduate. Without a company, the project couldn't continue. We also clearly understood that building such a large piece of infrastructure for industry, while continually bringing research ideas and the work in our papers into industry, was far beyond what an open-source project or an academic group could do on its own. We had to raise investment, start a company, and get industry involved. Starting a company was necessary. But moving to Silicon Valley—that's a misconception. Honestly, for the first six months, we didn't necessarily need to move here. We just had this impression that starting a company meant coming to Silicon Valley. Hahaha. So that wasn't a choice we absolutely had to make. The first decision, starting a company, was necessary.

Junchen Jiang (Guest) (01:23:41)

The second wasn't necessarily so.

Uncle Moon (Host) (01:23:45)

But now that you've been in Silicon Valley for over a year, do you think you could have brought the company this far without being here?

Junchen Jiang (Guest) (01:23:51)

I'm only saying we didn't have to be here for the first six months. Back then, there were fewer than seven or eight of us in total. It was just the founding team, all University of Chicago students and me. We really didn't need to come for those first six months. We came too early. But now, we have to be here. At our current stage, if you want good hires, good customers, and access to the right information, you have to be in Silicon Valley.

Uncle Moon (Host) (01:24:13)

Then you founded Tensormesh, a company built around LMCache, the KV cache storage system.

Junchen Jiang (Guest) (01:24:21)

Yes.

Uncle Moon (Host) (01:24:21)

Your website's slogan is essentially one sentence: don't keep paying for models to recompute the same content.

Junchen Jiang (Guest) (01:24:31)

Yes, that's right.

Uncle Moon (Host) (01:24:32)

Could you explain the thinking behind the company and the product in more detail?

Junchen Jiang (Guest) (01:24:37)

The thinking behind it, or our vision for the future, isn't quite the same as that slogan. Honestly, the slogan expresses a vision people can understand today. Hahaha. Our real vision is something larger. But what people can most readily understand now is whether our technology can save them money, make things faster, and let them serve more people with fewer resources. That's why we use that line. At the current stage, the tangible benefit KV cache technology offers industry is reducing repeated computation: preventing models from doing the same computation over and over on GPUs. If you save the result in KV cache form, you don't have to compute it again next time. For example, suppose you ask a model to read a book and answer a question, or read a report and answer a question. Why should it have to read the whole thing again the next time it sees that book?

Junchen Jiang (Guest) (01:24:37)

It could simply take the information it has already read— and save it as KV cache, a file containing the model's own internal memory. Wouldn't that be better? Right? Then the model wouldn't forget content it had already seen. That's why we say that one tangible benefit of KV cache, or of our company today, is reducing or avoiding repeated computation.

Uncle Moon (Host) (01:26:13)

You avoid repeated computation, but KV caches are very large, so they also take up a lot of storage. And storage is so expensive now. How do you weigh that cost trade-off?

Junchen Jiang (Guest) (01:26:26)

First, as I said, storage prices are temporarily inflated. That's because supply and demand are out of balance. Once production capacity catches up, prices will naturally fall. But you're right: storing more KV caches and more data like them puts greater demands on storage. That's why a new supply-demand relationship emerges. Once suppliers increase their capacity, prices will come down. But demand will definitely grow. More companies and products are storing the KV caches generated by the LLMs behind them, and they'll keep storing more. That's a very clear trend.

Uncle Moon (Host) (01:27:15)

Could you walk everyone through the whole process, in simple terms—from a user entering the prompt they want to send, through the KV cache and the GPU, to the final result? Explain the costs and exactly what you can optimize.

Junchen Jiang (Guest) (01:27:35)

Imagine you're a very smart programmer. I give you a program to analyze and ask whether you can add a new feature, write some new code. The first thing you have to do is read the program. A model is the same: it has to read and understand the program's contents. A person understands the program as structures they can retain in their mind, as a memory. The model has a memory too, and that's the KV cache. It's a collection of floating-point numbers, lots and lots of numbers. It's a matrix, a three-dimensional matrix. But that's how the model remembers new knowledge—things it hasn't seen during training. The new program I give it is something it hasn't seen. Before generating the new feature you've requested, it first has to read the program, then your question, and then generate the result.

Junchen Jiang (Guest) (01:27:35)

Next time, if I ask you to— add another new feature to that program, and you've forgotten your memory of reading it—which is what many models do now—you have to read it all over again. Suppose, in that LLM's workload, 90% of the input is the program. That's very plausible; it could be more than 90%, because the user's actual question is short. If 90% is the program and I save the KV cache, I'm avoiding 90% of that computation. I'm skipping that 90% of computation the model would have done again. It can simply load its memory back in, straight into its brain.

Uncle Moon (Host) (01:29:42)

So your company has always treated KV cache as an LLM's memory and managed it on that basis.

Junchen Jiang (Guest) (01:29:48)

Exactly. It is the model's memory. Today, people think the benefit of saving that memory is being able to reuse it. But think about it: if what you're saving is memory, what else can you do beyond reuse? You can understand what's inside it, what the model actually remembered, which tokens receive more attention, and which it considers more important. You could even optimize its memory so its future behavior improves, and so it generates better results when it receives that memory again.

Uncle Moon (Host) (01:30:23) †

So it isn't just a KV cache storage—

Junchen Jiang (Guest) (01:30:28) †

Right.

Uncle Moon (Host) (01:30:28) †

company. It could become a company that helps people understand how large language models themselves work.

Junchen Jiang (Guest) (01:30:34)

Exactly.

Uncle Moon (Host) (01:30:35) †

And improve how efficiently they work.

Junchen Jiang (Guest) (01:30:38) †

Yes.

Uncle Moon (Host) (01:30:38) †

Improve how that memory works—a company along those lines.

Junchen Jiang (Guest) (01:30:41)

Exactly. That's a vision, our long-term vision. But a company has to do business, right? First, we need to solve people's immediate problem: token costs are too high. Everyone is burning through tokens. How can we let them do that without spending so much money or so much compute, and serve them faster? KV cache is a way to help directly with that. But we believe it will also enable more things in the future that people haven't thought of yet.

New Inference Chips, Prefill, and the Future of KV Cache

Uncle Moon (Host) (01:31:14)

Now some computing paradigms may be changing, and new computing chips are emerging: Cerebras, which recently went public; Groq, which NVIDIA acquired; and things like the recently introduced LPUs. Do you think these new chips could affect the KV cache optimizations you're doing in software? Before answering that, could you explain their biggest advantages over today's GPUs, and what impact they might have on KV cache?

Junchen Jiang (Guest) (01:31:52)

My understanding—and of course I'm not a hardware person, so I can't be too categorical—is that most hardware innovation today focuses on more efficient decoding, more efficient generation. Most of our optimization focuses on analyzing the input more efficiently. That's called prefill. Remember that term: prefill. It's the process through which a model understands a long text input. Decoding is what happens after it has understood that input: generating tokens one at a time, producing characters or words one by one. Many of these new chips optimize decoding because that's the part most users can see directly. You're waiting for the model to generate an article, watching the words appear one by one. But people often overlook something: from the user's perspective, it seems the model spends 90% of its time decoding, generating the output— but actually, 90% of the compute goes to prefill, to processing the input.

Uncle Moon (Host) (01:33:11)

Is there data to support that?

Junchen Jiang (Guest) (01:33:12)

There's no direct statistical evidence for that. But if you look at agent traces and analyses, you'll see that agents have much, much longer inputs than outputs. An output might be a few thousand tokens, or at most a few tens of thousands. An input might be hundreds of thousands or millions of tokens. Any output can later become input. Today's output will always become a later input. That's why, if you really look at how GPU compute is allocated, most of it goes to the input. But output is important, and output speed is very important, because it's what people actually experience. If I'm waiting for an article and the output is slow, I'm unhappy. I don't care how quickly you processed the input or how much input you handled. From an infrastructure perspective, though, most compute goes toward processing the input. That's a widespread and very interesting misconception.

Junchen Jiang (Guest) (01:34:24)

People now see chip companies optimizing output and assume that output must account for most of the compute. But output isn't necessarily where most of the compute goes.

Uncle Moon (Host) (01:34:32)

That's a very interesting point. It's a little counterintuitive.

Junchen Jiang (Guest) (01:34:36)

Very counterintuitive.

Uncle Moon (Host) (01:34:37)

In that case, could we end up in a future where either OpenAI or Anthropic dominates, and its API prices keep falling as software and hardware improve? Then everyone can just use that API as easily as turning on a tap. What role would KV cache play in that world? Would it simply disappear?

Junchen Jiang (Guest) (01:35:03)

First, that's possible. The premise you're describing could happen. With any technology, one company might dominate or even establish a monopoly. But I don't think that even if one company, or a handful of companies, monopolizes the field, their services would be worthwhile for everyone. They might satisfy 80% of users' needs, but who serves the remaining 20%? It's like video streaming. There are countless video companies around the world. But if you look at the distribution of video traffic, most of it is YouTube, followed by Netflix—excluding adult sites, of course. Hahaha. Traffic within an industry can be very unevenly distributed. A few companies have enormous traffic, but there are still many companies that need to— They need to serve their own users. It's just like video, right? Nobody would say that HBO's video streaming isn't important. It is very important, but it serves only HBO's own users. AI models could develop in the same way.

Junchen Jiang (Guest) (01:36:15)

Even if Anthropic and OpenAI grow as big as YouTube and Netflix, there could still be plenty of AI providers, just as there are video services whose names everyone knows. Your traffic might not be as large as YouTube's or Netflix's, but you could still be very important. I think that's one possible direction. Whichever direction things take, though, let me return to your second question: how does this affect us? What's interesting is that what we're building isn't directly or tightly tied to the models themselves. We're addressing a problem that occurs with any model. Any model has a KV cache. At least all the large language models we can see today, including multimodal models, use a Transformer architecture. Of course, they might combine a Transformer with a linear model— A hybrid model. But it still has a KV cache.

Junchen Jiang (Guest) (01:37:25)

If every model needs to store its KV cache, it needs a separate layer that can take it from different hardware and different inference engines, and store it in different storage systems.

Uncle Moon (Host) (01:37:49)

So what you want to do is completely decouple the KV cache layer from all the other storage media and all the models, and turn it into an independent component?

Junchen Jiang (Guest) (01:37:59)

Right. Of course, we're not trying to decouple it from the model itself: the KV cache is still generated by the model, after all. What we want to decouple it from are the other components: the runtime environment, for example, the storage structure, the storage hardware, and the GPUs. It can be decoupled from all of those. That's the direction LMCache is taking. Whatever inference engine, storage system, inference architecture, or runtime architecture a system using large language models has, it should be able to store its KV cache in LMCache. LMCache becomes a shared layer.

Uncle Moon (Host) (01:38:36) †

Here's a fairly pointed question. Once you've decoupled this from the inference engine, the model itself, and the storage medium, how do you convince users that they need your product? Why wouldn't I simply use the KV cache built into the inference engine, or some kind of KV cache that comes with a cloud service?

Junchen Jiang (Guest) (01:38:54) †

Right.

Uncle Moon (Host) (01:38:54) †

Right? Why do I have to pay for your service?

Junchen Jiang (Guest) (01:38:56)

First of all, open source gives people choices. That's one of the biggest things open-source users want: I can use your product, or someone else's, and I can switch at any time. I have choices at every layer. I can choose this engine, that storage system, or that GPU. It's true that if I were serving just one user and optimizing their current system, knowing that it would never change, there would be no need to decouple anything. A tightly coupled system would be fine. Take Apple: it builds everything itself, from the chips to the applications. But the vast majority of companies don't operate that way or have that philosophy. So there really is strong demand for a highly decoupled KV cache layer. That layer has to perform well enough across all kinds of scenarios, though, so people don't feel that decoupling has a major impact on performance or support.

Uncle Moon (Host) (01:39:56)

So, as things stand, is that an underlying reason you believe you can differentiate yourselves and keep building this?

Junchen Jiang (Guest) (01:40:04)

Yes, it's an important reason at this point. Of course, it's not easy. It's something that really requires the community to work together. For example, to work with different chips and storage media, you have to collaborate with the storage vendors. We can't write all that code ourselves. We can't implement every integration. Engineers from those partner companies have to work on it with us for it to succeed.

Uncle Moon (Host) (01:40:34) †

So, at a deeper level, you're essentially creating an industry standard for KV cache, and you want everyone to help build it?

Junchen Jiang (Guest) (01:40:42) †

Right.

Uncle Moon (Host) (01:40:42) †

If one company is more willing to integrate with your standard, other model providers might then be more inclined to use that company's storage medium or inference engine.

Junchen Jiang (Guest) (01:40:55)

Right. That's one of the motivations for many of the companies working with us now. They feel that integrating with LMCache opens up more possibilities, including opportunities to work with more vendors.

Uncle Moon (Host) (01:41:11)

We've seen quite a few successful examples of that in Silicon Valley: growing a community and an open-source project until it becomes a de facto standard.

AI Agents and Model-Native Memory

Uncle Moon (Host) (01:41:19)

Another topic that's getting increasingly popular is coding agents. You mentioned earlier that you're managing memory for large language models. If I imagine a coding agent as an individual living inside a GPU, my understanding is that you're effectively giving each agent its own memory-management system. And it's model-native memory management: you're storing memories that the model can understand and use directly. In that setting, what greater value can you provide compared with an ordinary chatbot?

Junchen Jiang (Guest) (01:41:55)

First, agents are very interesting. Their capabilities don't come simply from the model itself being more powerful. Rather, as they interact more with their environment, they become more adaptable.

Uncle Moon (Host) (01:42:14)

That does still come partly from the model's capabilities, though.

Junchen Jiang (Guest) (01:42:18)

Yes, absolutely. The model has to support it. But if you use the model only once, you aren't making use of that capability, right? An agent effectively maximizes it. In doing so, though, the number of interactions with both the environment and the model keeps increasing. The model also has more and more to remember over the course of completing an entire task. And remembering it becomes more complicated. It's not simply a matter of remembering everything. It has to keep shuffling things around, retrieving the most relevant memories, figuring out which memories are relevant right now, and deciding what should come first and what should come later. The more interactions there are, the more memories there are. The more complex the task, the more complex the memory-management process needs to be. All of that ultimately lands in the memory-management module. Today, most memory-management modules manage memory in a form that humans can understand. They manage— Human language: text, code, and images.

Junchen Jiang (Guest) (01:43:28)

What we manage is the counterpart of that text and human language in a form the model can understand. You can think of it as translating human language into memories that the model can read directly, so you can manage the model's memory itself.

Uncle Moon (Host) (01:43:49)

OpenAI has been publishing related blog posts recently. They're now able to offer better memory mechanisms. As you use OpenAI more and more—

Junchen Jiang (Guest) (01:43:56)

Right.

Uncle Moon (Host) (01:43:58)

It can produce responses that are better suited to you, that kind of—

Junchen Jiang (Guest) (01:44:01)

Yes. Honestly, I find that quite annoying. Quite annoying. I keep telling ChatGPT, 'Forget what I told you before.' Sometimes I discuss company matters with ChatGPT. [Laughs.] Then sometimes I'll ask it about something completely different: what advice does it have on this other thing? And suddenly it brings up Tensormesh. [Laughs.] I say, 'That has absolutely nothing to do with this. You're bringing up an old memory.' You can see that ChatGPT is trying to personalize your memories and how those memories affect subsequent interactions. But sometimes it's pretty annoying. [Laughs.]

Uncle Moon (Host) (01:44:45)

What you were just describing is managing the model's own memory.

Junchen Jiang (Guest) (01:44:50)

Right.

Uncle Moon (Host) (01:44:50)

Compared with managing the user's prompts or past inputs directly at a higher layer, what do you think is the biggest advantage?

Junchen Jiang (Guest) (01:44:56)

Think of it this way. Broadly speaking, there are three kinds of AI engineers. The first are the people who actually train models. They think the way they change the world is by changing model weights: they train a new, better model, and everyone uses it. Another group builds applications and agents, writes prompts, and does context engineering. They think their way of changing the world is: give me the best model, and I'll change its context and its inputs to get the most out of its potential. We sit in between. We're taking a third path. What is that path? We look directly at the memory generated inside the model by an already-trained model and an already-written application. We try to understand, optimize, and store that memory. And this approach— It has three benefits. It can improve system efficiency, give you insight into what's happening inside the model—what it's thinking about and what it's paying attention to—and also change the model's behavior.

Junchen Jiang (Guest) (01:46:07) †

If you tamper with it—actually, 'tamper' isn't a very good word. If you optimize the KV cache, you can change the model's behavior. You can make it look in the right places and produce the right results, instead of looking at everything and producing an average result.

Uncle Moon (Host) (01:46:37) †

As you were saying that, a picture came to mind. There's a very popular term lately: self-improvement, or agents evolving on their own.

Junchen Jiang (Guest) (01:46:46) †

Right.

Uncle Moon (Host) (01:46:46) †

As users and agents interact more and more frequently—or perhaps the agents themselves can be used to improve the next generation of models—you mentioned directly modifying the KV cache, directly modifying the model's memory. Have you thought about using this within that kind of self-improvement framework?

Junchen Jiang (Guest) (01:47:10)

You're right. Exactly. Self-improvement, personalization, and things like that are changes in the paradigm, or more precisely, changes in the objective. You give AI a new objective, whether it's self-improvement or personalization. Those are changes in the goal. With a new goal, you have the three paths I just described: train the model; change the agent application or the prompt; or change the KV cache. I think those three paths complement one another. So yes, this is a new direction, and KV cache can be one route toward it. But there's something I may not have emphasized clearly enough. A lot of what I'm saying probably sounds very strange to most people. In most people's eyes, the KV cache is just temporary data generated while a model is producing output, something you throw away once generation finishes.

Junchen Jiang (Guest) (01:47:10)

Many people don't think of it as—people who really understand model training, who train the models— To them, KV cache is like GPU CUDA: something they don't see and don't care about. Most people working on inference also see it as something the model generates, an indecipherable artifact that nobody can understand. But to us, KV cache is gold. It's the oil of the future. Right now, people might think: this isn't like prompts or prompt engineering, where you can directly read and understand what's there. Nor is it like model training, with so many companies backing it and proclaiming that it's the future. Right now, Tensormesh is the only one saying this. To be honest, we're the first to do it, the first to venture into this territory. But we see it very clearly: after model optimization and prompt optimization, KV cache will be the next biggest opportunity. The data itself is extremely important.

Uncle Moon (Host) (01:49:14)

So you feel that all KV cache that can be stored should be stored, and then used to improve the next generation of models or the next paradigm?

Junchen Jiang (Guest) (01:49:22)

You could put it that way, although that might be a little extreme. But we do believe that any context people consider important—any important context—should be represented as KV cache.

Uncle Moon (Host) (01:49:35)

Anyway, we talked earlier about Steve Jobs and his reality-distortion field. [Laughs.] In a world like this—

Junchen Jiang (Guest) (01:49:41)

You think I'm brainwashing people, don't you?

Uncle Moon (Host) (01:49:43)

Yes. Whatever the case, if you believe in something in this world and don't try to bend reality by starting a company, life would be rather dull, wouldn't it?

Junchen Jiang (Guest) (01:49:53)

Right, of course. But let me put it this way: everyone sees it differently. If you read the Steve Jobs biography, most references to his reality-distortion field are actually negative. Most people regard the term as pejorative. To be honest, Chinese people call it 'XN'—brainwashing—while Westerners call it a reality-distortion field. In plain Chinese, it's brainwashing, which isn't necessarily a good thing. Still, what we're working on is something the industry hasn't acknowledged yet, or something we believe the industry is about to acknowledge but hasn't so far. That takes some courage. Starting this company and using what little influence we have to tell people that this is genuinely important is a big challenge.

Uncle Moon (Host) (01:50:39)

Then I'll contribute my own small effort by recording this entire conversation online. [Laughs.] If it really does come true, people can come back and watch it again and again. Here is someone who still believes in a reality-distortion field and keeps working toward making it happen.

Fundraising, Industry Consensus, and Racing Against Yourself

Uncle Moon (Host) (01:50:54)

In this funding round, you received investment from NVIDIA, AMD, and other companies, some of which are competitors. How do you think you won them over?

Junchen Jiang (Guest) (01:51:07)

That's a good question. I used to think industry was a black-and-white world: if companies competed, they must dislike each other. Honestly, they do dislike each other. But even while they do, they can agree on certain technological trends. What's interesting is that each of those two companies independently sees tremendous potential in KV cache. At least, NVIDIA's interest in our technology had nothing to do with AMD, and AMD wasn't interested because NVIDIA was interested. They independently believe that a really well-designed KV cache layer can be a key factor in getting the most performance out of their hardware. So both want to invest in KV cache. Their own people will work on it, and they'll work with us as well. I'm not saying that because they invested in us, they'll stop doing it themselves. They'll still work on it internally. But they believe this technology has to work: if it isn't done well, they're in trouble too.

Junchen Jiang (Guest) (01:52:18)

So they have to invest in the best companies working in this field. Right now, I'd say we're probably the best company working on KV cache, partly because the field isn't very large yet. [Laughs.]

Uncle Moon (Host) (01:52:36)

It seems you might be the only company in this field. [Laughs.] Right? So this is probably the first time you've publicly discussed both companies investing in you without knowing the other was investing—

Junchen Jiang (Guest) (01:52:48)

At some point, they did find out. But when they first approached us, neither knew about the other. Of course, perhaps someone internally knew: other people were working with Tensormesh, so AMD should work with them too, that sort of thing. But that wasn't why they approached us. I really think they believe they need good KV cache technology—storage, transfer, and optimization—to get the greatest efficiency from their hardware.

Uncle Moon (Host) (01:53:19)

How much time passed between your previous funding round and this one?

Junchen Jiang (Guest) (01:53:22)

We've indeed raised two rounds, with only about eight months between them.

Uncle Moon (Host) (01:53:27)

Over these past eight months to a year of building the company, what has been the biggest difficulty?

Junchen Jiang (Guest) (01:53:31)

The biggest difficulty is finding the strongest possible motivation to keep moving when you don't have a direct competitor. In academia, we always had paper deadlines. Industry doesn't have paper-submission deadlines, right? Of course, competition can provide motivation. Think of the two well-known giants in inference engines: they certainly have motivation. We don't have that sort of powerful external pressure driving us. There is some activity, of course. Some teams and engineers inside large companies work on this to justify their own value. But they aren't doing it as direct competitors to us. That's the biggest challenge for me: how to race against ourselves. We all know time is very tight, though. KV cache hasn't yet reached its most important moment. People haven't yet realized just how important KV cache is. Once they do realize it, every big company will rush to work on it.

Uncle Moon (Host) (01:54:54)

You're that certain?

Junchen Jiang (Guest) (01:54:55)

I absolutely believe it. I'd bet you on it. It's really true. At some point—perhaps in less than a year—every company, or every big company, will introduce KV cache products. By then, we have to be able to stand alongside them, at the very least, or be ahead of them so they can't catch us. We've always known that pressure is there. But it's different from a paper deadline. A paper deadline sits there and says: miss this date, and you can't submit. Here, there is no deadline like that. Coming from academia into a startup, I've found that you really need a different way to motivate everyone.

Uncle Moon (Host) (01:55:35)

Then let's keep going. We can make that bet, and later we'll see what actually happens. So the greatest difficulty you've encountered has been figuring out how to race against yourself.

The Shift from Academic to Founder

Uncle Moon (Host) (01:55:44)

Compared with your more than ten years as an academic, what do you think has been the biggest shift or improvement in your thinking?

Junchen Jiang (Guest) (01:55:50)

There have been so many. We're learning that whole world of industry for the first time. It feels like being thrown straight into the ocean to swim without ever having trained in a pool. Building a startup requires a different set of skills from surviving at a large company, and possibly more skills. It's extremely difficult, so we've learned a tremendous amount. Sometimes I tell my academic friends that, in some respects, I've learned far more in these past two years than in my previous fifteen years in academia. One particularly interesting thing is that in academia, you may have some contact with industry, but you never meet investors. Investors were an entirely new species to me, an entirely new kind of person to deal with. I'd never talked to one. What do they want? What do they care about? Fortunately, Silicon Valley has so many investors that I could practice by talking to them one by one.

Junchen Jiang (Guest) (01:55:50)

If there had been only two, I really wouldn't have known what to do. Once I'd missed those two opportunities, there would have been nobody left. I remember my first experience with an investor very clearly. Peter wasn't even someone I had contacted myself; Hui Zhang connected us. I met him several times without realizing that he was evaluating me. After a few meetings, Professor Zhang said, 'No, no, we need to discuss how you should pitch this.' [Laughs.] A couple of days later, Peter did contact me again, and we had another conversation. Then I didn't hear from him. At first I thought that meant he wasn't interested. But he kept coming back every two weeks, and I wondered whether he was interested or not. Investors were a kind of creature I'd never encountered before. Once, he even flew to Chicago with a manager who worked with him, saying he wanted to talk with me and the other co-founders. I thought, 'How interesting.'

Junchen Jiang (Guest) (01:57:01)

He said he was passing through Chicago on his way somewhere else. I said, 'Fine, let's meet at a coffee shop.' I went to the coffee shop near the university. On my drive to campus that day, I called Hui Zhang and said, 'Professor Zhang, this Peter—' 'Why has he flown here?' Professor Zhang asked, 'How are you planning to talk with him?' I said, 'I'll find a coffee shop.' He said, 'Come on, that's far too informal. You're just going to chat in a coffee shop? At least give them a presentation. Find a room and present to them.' I thought, 'Oh no, I had no idea what he'd come for.' [Laughs.] But Peter understood that we were professors starting a company for the first time and didn't know how investors operated. He was very patient and kept talking with us. I wasn't talking to anyone else at the time, so he wasn't in a hurry either.

Junchen Jiang (Guest) (01:58:14)

When he actually made an offer and gave me a term sheet, I suddenly thought, 'Shouldn't I find out what is actually in this term sheet?' That night I called Hui and a lawyer to ask how to interpret it and how to negotiate. Everything was new. Dealing with investors is a trap many professors have fallen into. Later, lots of professors came to me, because many people in computer systems are starting companies now, and they knew I was doing it. They asked how I had learned all this. I said, 'There's really no particular way to learn it—' The investors actually know that we aren't good at fundraising.

Uncle Moon (Host) (01:59:29)

[Laughs.] So why do you think they ultimately invested in you?

Junchen Jiang (Guest) (01:59:32)

Honestly, investors tailor their approach to the person. They know you don't understand this world, so they don't put you through all the usual games. If you've raised lots of money and built many startups before, they won't forgive you for making elementary mistakes. But you're a professor who's never done this. A few basic mistakes don't matter, as long as what you're building is real—as long as they see gold in it and can see its future. I think investors do have that judgment, at least those who have invested in and worked with academics.

Uncle Moon (Host) (02:00:10)

So if your work is good enough, it doesn't matter whether you know all those tactics. Investors can make their own judgment.

Junchen Jiang (Guest) (02:00:17)

Investors have their own judgment. At least the ones who've worked with academics do. They've seen people like us before.

Starting a Company with Students: Trust and Decisions

Uncle Moon (Host) (02:00:25)

After raising the money and starting the company with your students, what struck you most about the transition from a teacher–student relationship to being co-founders?

Junchen Jiang (Guest) (02:00:36)

First, some things really haven't changed. I have a view that many professors may find a bit strange: one major reason professors' startups fail is that their students didn't join them. People can see one benefit of the teacher–student relationship: it's a very stable structure. The professor has a vision, everyone trusts that vision, and the professor trusts the students' ability to execute. But what people may not realize is how enormously that structure benefits a company's development, especially early on. The advantages can be greater than you'd imagine. Honestly, if I had started this company with a friend, it wouldn't have gone nearly as smoothly. We wouldn't already have learned how to work with each other's strengths, and those strengths might even conflict. Starting a company with your students gives you an inherent advantage in that respect. Of course, many professors will say, 'I didn't start a company with my students either.

Junchen Jiang (Guest) (02:01:47)

My students didn't want to do a startup, so I went out myself, found a good professional CEO, and it went very well.' Those examples certainly exist; there are plenty of them. But if you genuinely want to build a technology startup, I think doing it with your students is an excellent choice. I've seen quite a few people around me with similar backgrounds—I won't name names—who started companies without bringing their students along, and they ultimately didn't succeed.

Uncle Moon (Host) (02:02:16)

One hundred percent?

Junchen Jiang (Guest) (02:02:16)

Almost. That's just my small sample, though, so it doesn't have any statistical significance.

Uncle Moon (Host) (02:02:21)

During that process, you must make a lot of decisions. Have there been times when your students all disagreed with you, and perhaps in hindsight they were right and you were wrong?

Junchen Jiang (Guest) (02:02:33)

Yes, absolutely. But I come back to what I said: that overall structure is beneficial. Even if, in the short term, they tell me, 'Junchen, your idea is too ambitious. Technically, we can't do it yet,' I'll trust them. If they say it's not technically feasible, I believe their technical judgment. When they push back on technical grounds, the premise is still that they believe the direction is right, but it can't be done with the technology right now. That trust is there. They aren't using a technical excuse because they don't believe in your vision; they genuinely think it's not possible technically. And I trust that this is genuinely their view, that we can't do it yet, but can revisit it later. That's why I think the teacher–student relationship matters. That said, now that we've started a company, the three of us are equals. There are three co-founders, but when we started, almost all the students received founding equity.

Junchen Jiang (Guest) (02:03:45)

We were all building it together. It's just that those three students had graduated—those two students had graduated and could join me full-time as co-founders. The other students were contributing a lot as well. So, as you said, once you start a company, the relationship changes. We become equals, colleagues. But because we've worked together before, the trust is already there. I trust their technical abilities, and they trust my vision.

Uncle Moon (Host) (02:04:12)

In this age of AI and agents, trust between people has actually become one of the most precious things.

Junchen Jiang (Guest) (02:04:19) †

That's how I feel too. When nobody trusts agents, people end up helping one another really well.

Uncle Moon (Host) (02:04:25)

Often, you can only keep going for the long term if you really trust one another. As the saying goes, one person can go fast, but a group can go far.

Junchen Jiang (Guest) (02:04:33)

Yes, I believe that very, very strongly.

OpenAI's API and the Internet's Shared Standards

Uncle Moon (Host) (02:04:35)

Recently, you've been sharing an argument on social media that OpenAI's API is the IPv4 of this new era. From a networking-systems perspective, could you explain that idea in more depth? If they're IPv4, what are you?

Junchen Jiang (Guest) (02:04:55)

Let me put it this way. I genuinely find that argument interesting. It doesn't necessarily have a direct connection to our company's strategy, but the argument itself has deeper implications. As I said, AI has just had its Internet moment. Think about when the Internet took off in the late 1990s, and all the architectural changes over the following twenty years. Those changes in pricing ultimately helped turn the Internet into a thriving global system with a rich ecosystem. Along the way, certain parts had to gain broad recognition as important and settle into stable components, or nodes in the layered architecture. When we studied networking, there was this funnel-shaped structure with a narrow waist in the middle. That narrow waist is the IP layer. The IP layer is something every application— Recognizes as a communication interface. Innovating at the IP layer, within that protocol, is extremely difficult. There was something called IPv6; I don't know whether everyone is familiar with it.

Junchen Jiang (Guest) (02:06:08)

Everyone thought IPv6 was much better than IPv4, the earlier IP protocol. It could support more optimizations and more applications. But ultimately it didn't succeed, because everyone was already using IPv4. Why bring up that history? When a new ecosystem is growing wildly, in its initial period of rapid growth, some structures inevitably become stable. People in industry realize that without fixing those structures, none of them can move forward. When I say the OpenAI API, I don't mean the particular API OpenAI itself provides. I mean the OpenAI-API-compatible format, the query format. That is what all these providers—the application providers— The model providers and the inference-service providers—all recognize as a shared structure that has stabilized in the middle. When I made that comparison, I wanted to compare today's AI ecosystem, its overall framework, with the Internet's framework after more than a decade of development. AI's framework is maturing very quickly. Consensus has already emerged around certain parts.

Junchen Jiang (Guest) (02:07:18)

In the future, we'll see more protocols, stable structures, and data formats like the IP layer or the OpenAI-API-compatible format. More of these will develop. KV cache could be the next one, or the one after that. Right now, there aren't many people working on KV cache, or trying to reuse it or transfer it between services— There aren't many people transferring KV cache yet. But as these services multiply, KV cache will become a layer like that too.

Uncle Moon (Host) (02:08:39)

Returning to AI infrastructure, you've kept emphasizing the importance of KV cache becoming an independent layer. Apart from KV cache, is there anything you think is particularly underrated today but could attract a great deal of attention later? Just as with the past twenty-plus years of the Internet, many treasures only get discovered further down the road.

Junchen Jiang (Guest) (02:09:00)

Yes, yes. A lot of people think KV cache has something to do with a key-value store, but it doesn't. It's about two tensors, two large matrices. It has nothing to do with a traditional database's key-value store.

Uncle Moon (Host) (02:09:15)

Got it.

Junchen Jiang (Guest) (02:09:15)

A lot of people hear my talk and ask, 'How does this relate to key-value stores?' I think, 'Oh no, everything I said was wasted.' [Laughs.] But okay, back to your question. I think there's a blind spot. Hardware providers are following a trend that I call 'IBM-ification.' What do I mean? To maximize their returns and their margins, they bundle everything together and sell it as one mainframe or large piece of equipment: processors, networking, storage—everything sold together.

Uncle Moon (Host) (02:10:00)

Oh, I know who you're talking about.

Junchen Jiang (Guest) (02:10:02)

Right. But think about it: that's following IBM's old path. I'm not criticizing IBM itself. These are all large, excellent companies. But if you study history, you'll see that mainframes ultimately didn't become the dominant form of data-center computing. Why? Technically, a mainframe is neither the cheapest nor the most flexible approach. If you look at the big banks in the 1990s, they all bought mainframes. HP and IBM built them those machines; EMC and others were also doing that. But what ultimately became the platform for large-scale computing? Cheap memory, cheap storage, cheap CPUs, and cheap networking, connected in the smartest possible way. That became the data center. After standardizing the components, you combine small pieces intelligently. I think that is the direction of history, and it's something that will happen sooner or later. That doesn't mean the big companies taking this path today—the very largest companies—will disappear. They'll still exist.

Junchen Jiang (Guest) (02:11:13)

But I think competitors will eventually emerge with smaller, more flexible, more scalable systems.

Uncle Moon (Host) (02:11:33)

So you think these possibilities may gradually be uncovered next?

Junchen Jiang (Guest) (02:11:37)

Possibly. You may have heard the term 'disaggregation,' a disaggregated architecture. Taken to its extreme, it means specializing each component and making it replaceable at any time. The real core, the hardest part, is how you combine them. If you can solve that, a new architecture could emerge, a new way of organizing AI infrastructure that could be disruptive. So if you're asking what I find promising besides KV cache, that's promising.

Uncle Moon (Host) (02:12:16)

Then we'll wait and see.

Choosing Models, Applications, or Infrastructure

Uncle Moon (Host) (02:12:18)

Many undergraduates and graduate students feel it's getting hard to find a job. What advice would you give them? Should they keep betting on large models, applications built on those models, or the infrastructure for them?

Junchen Jiang (Guest) (02:12:32)

First, I think large models remain very important, with a great deal of room for development. AI itself is not a solved problem. Everyone knows about AGI. The current large-model path is very promising and has room for improvement, but a completely new path could also emerge. So if you want to work on models or study them, don't just keep doing the kinds of models everyone has already worked to death. Working on new models is a good direction. Of course, I'm speaking to researchers here, to PhD students. For infrastructure, industry is the best place to go. But you shouldn't go into a role that is purely about writing code, because AI can now replace so much of that. What is hard for AI to replace? People's ability to take responsibility. I don't mean simply being brave enough to say, 'AI wrote this code, and I'll guarantee it and maintain it.' That's not what I mean.

Junchen Jiang (Guest) (02:12:32)

I mean having the ability— To give structure to the process of having AI write code for you. Put simply, it's what we call taste in English: design taste. You know which designs are good, which will last, which are easier to maintain, and which are more flexible. Of course, AI can already offer advice on these things. But if you don't know for yourself what's good or bad, it's like AI giving you a picture: if you can't judge whether the picture is good, how can you sell it to someone? If your design depends entirely on AI, you won't even be able to maintain your own code. That's something people working on AI infrastructure need to think through. Your last option was applications, right? I don't have much authority to speak about that. Honestly, I find the current situation with applications very difficult to assess. On the one hand, the prospects are enormous.

Junchen Jiang (Guest) (02:13:45)

There are so many things that currently use— Traditional business models, and AI could disrupt all of them. But after you disrupt them, Anthropic might turn around and disrupt you two days later. It's really hard to say. I find it extremely difficult to predict.

Uncle Moon (Host) (02:15:10) †

Then we're just discussing possibilities. It's hard for anyone to say.

Junchen Jiang (Guest) (02:15:12) †

Exactly.

Uncle Moon (Host) (02:15:13)

You treat AI as the next generation of the Internet and build an app, but very soon, what that app does becomes a native capability of the Internet itself.

Junchen Jiang (Guest) (02:15:20)

Right. It's very hard to say.

Uncle Moon (Host) (02:15:22)

For you personally, if you could go back, which decision would you most wish you hadn't made? What would you most want to change?

Junchen Jiang (Guest) (02:15:31)

If I could choose again, I'd probably start working on AI infrastructure earlier. If I had recognized the potential of Transformers back then, I would have started sooner. Honestly, as I said, starting in 2023 was a little late. Fortunately, the students were strong. I didn't even know what KV cache was at the time. When I first saw it, I thought, 'This looks reusable. That's a breakthrough!' Then I discovered that apparently everyone already knew that. We were lucky, really lucky. If the students hadn't been so strong, and if we hadn't recruited excellent people from the beginning, both on the business side and the technical side, we wouldn't be where we are today. So if I could go back and choose again, I'd either tell myself that this— Is extremely risky. You have to think it through, because the probability of failure is very high. Or I'd tell an even earlier version of myself, 'Transformers are going to take off.

Junchen Jiang (Guest) (02:16:42)

Get started right away.' [Laughs.]

Uncle Moon (Host) (02:16:54)

When you were interning at Microsoft Research, there must have been a lot of outstanding AI infrastructure people there, right?

Junchen Jiang (Guest) (02:17:00)

At the time, AI infrastructure was all about training. This is something I've always believed: I've never believed training infrastructure was the future.

Uncle Moon (Host) (02:17:10)

Because it's too much of a one-off, right? It's not very sustainable.

Junchen Jiang (Guest) (02:17:14)

It's too far behind the scenes. Too far behind the scenes.

Uncle Moon (Host) (02:17:18)

Right. You want to work on something more visible, directly facing users—

Junchen Jiang (Guest) (02:17:21)

Right.

Uncle Moon (Host) (02:17:21)

Infrastructure.

Junchen Jiang (Guest) (02:17:23)

Yes, exactly.

Advice at Different Career Stages

Uncle Moon (Host) (02:17:25)

Then let's make this the last question—or really a series of questions. Let's offer different groups of people some reflections or advice, either is fine. First, what would you say to a twenty-two-year-old college student who's about to graduate?

Junchen Jiang (Guest) (02:17:41) †

Ah.

Uncle Moon (Host) (02:17:41)

Right?

Junchen Jiang (Guest) (02:17:42)

First, do what challenges you the most. I've heard quite a few people say that, and I strongly agree.

Uncle Moon (Host) (02:17:53)

AMD's Lisa Su says that all the time, right? The younger you are, the more willing you should be to take on challenges.

Junchen Jiang (Guest) (02:17:59)

She wasn't one of the people I heard it from. [Laughs.] But good, she's said it too, right? Don't worry initially about what suits you best, what pays more, or what gives you stability most quickly. At twenty-two, do the hardest thing, because that's how you learn fastest. To be honest, one reason I pursued a PhD was that, at the time, the best students went to the United States for a PhD. But a PhD is extremely difficult. When we were undergraduates, writing a SIGCOMM paper felt impossible. Nowadays everyone thinks it's easy, right? So do what genuinely challenges you the most.

Uncle Moon (Host) (02:18:57)

What advice would you give a twenty-six- or twenty-seven-year-old PhD student?

Junchen Jiang (Guest) (02:19:01)

Does a twenty-six- or twenty-seven-year-old have to be a PhD student?

Uncle Moon (Host) (02:19:03)

Let's start with PhD students.

Junchen Jiang (Guest) (02:19:05)

If they're already in a PhD program, okay. I'd say: think about whether you really want to become a professor. I'm talking about computer science here. If you want to become a professor in CS, you have to finish the PhD, and you still need to become one of the best people in your area. There's no question about that. But if you aren't set on becoming a professor, these days you should think about what kind of research will expose you to the cutting edge of industry. The relationship between industry and academia is completely different from what it used to be. Why do I say that? This may be a provocative view: there may now be too many PhDs, too many people with graduate training, in industry. That's our fault—the professors' fault, the universities' fault for producing so many CS graduates. [Laughs.] If there is genuinely something worth researching in industry, industry will work on it very quickly.

Junchen Jiang (Guest) (02:19:05)

So— If you aren't determined to become a professor and devote yourself to teaching and mentoring, think about doing work connected to the most advanced things happening in industry.

Uncle Moon (Host) (02:20:25)

So, if you're a PhD student who doesn't want to become a professor, you should get more exposure to industry.

Junchen Jiang (Guest) (02:20:29)

Yes, to the cutting edge of industry, not just industry for its own sake. If you go into industry merely to build a webpage, that's no use. It's really a waste of time.

Uncle Moon (Host) (02:20:39)

So if you can get access to cutting-edge work in industry, leave when it's time to leave. It's okay not to finish the PhD, right?

Junchen Jiang (Guest) (02:20:45) †

I didn't say that. [Laughs.]

Uncle Moon (Host) (02:20:48)

It's pretty close, though, isn't it? Okay. Industry has a lot of resources now, and with all those PhDs, it also has a substantial research culture.

Junchen Jiang (Guest) (02:20:57)

I don't think it's because industry has more resources. The main reason is that so many people with research backgrounds are there. If industry has a genuinely good problem, they can work on it very well. But there are some things that require settling down and concentrating for a long time, and industry certainly can't do those.

Uncle Moon (Host) (02:21:13)

What about someone who's twenty-five or twenty-six, already working in industry, and hasn't pursued a PhD?

Junchen Jiang (Guest) (02:21:19)

If they want to do a PhD, now is a good time. Why? Think about the biggest challenge for people in industry: sooner or later, AI will replace the work they used to do. You need to recharge, to think about what will differentiate you in the future, what makes you different from others. Perhaps you've earned enough money, and going back for a PhD is a very appropriate choice. [Laughs.] Really. Among our PhD applicants now, we see a lot of people coming from industry.

Uncle Moon (Host) (02:21:54)

So people who can't catch this wave, or who have already caught it and made some money, can return to academia and take their time thinking about the next—the next—

Junchen Jiang (Guest) (02:22:03)

Right, recharge. Honestly, you don't have to go into academia to do that. You can recharge at a startup too. One advantage of a startup is its powerful survival instinct: it has to build new things that can't be replaced. If you go there and build something new and irreplaceable, that is recharging.

Uncle Moon (Host) (02:22:24)

Right. Help yourself secure a place in the future.

Junchen Jiang (Guest) (02:22:27)

Right.

Uncle Moon (Host) (02:22:28)

That's excellent advice. If you're twenty-five or twenty-six and haven't pursued a PhD, you can apply for one. Or you can commit to a startup, pursue entrepreneurship, and take a shot at building your future.

Junchen Jiang (Guest) (02:22:37)

Right.

Uncle Moon (Host) (02:22:38) †

If you took that advice back to 2020 or 2021, AI seemed to be fading a little at the time, right? But OpenAI was actually gathering momentum, ready to take off.

Junchen Jiang (Guest) (02:22:50) †

Right.

Uncle Moon (Host) (02:22:50) †

If you'd chosen OpenAI back then, you would have taken off spectacularly.

Junchen Jiang (Guest) (02:22:56)

[Laughs.] Yes, but at that point, OpenAI hadn't actually placed its bet on GPT yet.

Uncle Moon (Host) (02:23:00)

Right, right. The earlier, the better!

Junchen Jiang (Guest) (02:23:02)

If you'd joined OpenAI to work on the reinforcement-learning projects that played games, you might well have ended up being laid off. [Laughs.]

Uncle Moon (Host) (02:23:09)

But with post-training, reinforcement learning is popular again. That experience is hard to replace.

Junchen Jiang (Guest) (02:23:14)

Yes, that's true. Joining a startup now, or joining OpenAI back then, are both very good choices, in my view.

Uncle Moon (Host) (02:23:22)

Finally, what experience would you share with people around thirty who are still early in their careers?

Junchen Jiang (Guest) (02:23:34)

By thirty, you have many more things to consider. It's hard to say, 'I should just take on the greatest possible challenge.' That might not even be responsible. But if you have the chance to do something big with like-minded people, that kind of experience is very rare. I'm not saying it has to be a startup. You could work inside a company on something other people don't necessarily value. In academia, you could pursue a direction that not many people believe in. Or you could start a company around something with genuine potential that isn't widely favored. You still have those opportunities in your thirties. Really, they're always there. But your thirties are a time when you've accumulated experience and may be able to devote yourself to something long enough to make it succeed. Honestly, with the students who started this company with me, the greatest pressure I feel is making sure they don't waste this opportunity.

Junchen Jiang (Guest) (02:24:44)

You know, they could all earn very high salaries.

Uncle Moon (Host) (02:24:47)

Absolutely. Those people are incredibly strong. I know their abilities very well.

Junchen Jiang (Guest) (02:24:50)

Yes.

Uncle Moon (Host) (02:24:51)

After today's discussion, I think everyone can sense that Junchen wants to push beyond the limits. Whether in his own life or in his advice to others, he wants to use the time available to take on more challenges and raise the ceiling of what's possible. That's been your consistent approach to life and work. Finally, thank you, Junchen, for joining us for this two- or three-hour conversation.

Junchen Jiang (Guest) (02:25:19)

Thank you.

Uncle Moon (Host) (02:25:19)

I hope everyone enjoyed it and will continue supporting Uncle Moon's channel. Thank you all.

Junchen Jiang (Guest) (02:25:23)

All right.

Uncle Moon (Host) (02:25:24) †

All right.

† At least one speaker change or brief response in this passage needs audio verification.