Junchen Jiang, University of Chicago professor and CEO of Tensormesh, on betting his tenure on ChatGPT, turning research papers into open-source infrastructure, and why a model should never read any input twice

Junchen Jiang is a professor of computer science at the University of Chicago, where he leads the lab behind LMCache, the open-source KV Cache layer for LLM inference that plugs into vLLM. He entered Tsinghua's Yao Class (its elite CS program) in 2007, then did his PhD at CMU under Hui Zhang, studying the video streaming systems Netflix, HBO, and YouTube were then building. In 2023 he turned the lab from video to LLMs, betting that a model's internal memory would become the next data type worth caching — LMCache is the result.

He is co-founder and CEO of Tensormesh, the company commercializing LMCache: it stores a model's KV cache — the internal memory a transformer builds while reading its input — so an LLM never recomputes content it has already read. The larger ambition is a caching layer for all of AI, independent of any inference engine, storage system, or GPU. Tensormesh came out of stealth in October 2025 and recently closed a $20M round with both Nvidia and AMD investing.

In our in-depth conversation, we discuss:

  1. From math competitions into Tsinghua's Yao Class
  2. Why he quit theory: he sped up an algorithm, a professor asked him to prove its ceiling, and he realized he didn't care
  3. 2010: Netflix, HBO, and the BBC all suddenly needed a video streaming strategy, and HTTP turned out to carry video
  4. Why three months writing a video player at Conviva helped his research more than three Microsoft Research internships
  5. The PhD courses that make you read classic papers are really history classes — and history is how he spots what's next
  6. January 2023: students ready to skip a SIGCOMM deadline for ChatGPT, and his call that this was AI's Internet moment
  7. The web got CDNs and big data got Hadoop; his bet is that AI will need the same for KV Cache
  8. Why a networking lab picked the KV cache: in 2023, the only AI data heavy enough to challenge a network
  9. Open-sourcing LMCache after industry asked what KV Cache storage system he meant
  10. The $20M round: Nvidia and AMD investing, and staying ahead before every big company piles in

My biggest takeaways from this conversation:

  1. Theory defines a system's capabilities and upper limits. Jiang wanted to push that ceiling.

    As a Yao Class undergrad who joined a networking group, he multiplied an algorithm's speed and took the result to a Yao Class professor, expecting excitement. The professor's first reaction: prove that under certain conditions the improvement could never exceed a bound. His own reaction settled his career: "I don't care what its theoretical ceiling is. What I care about is whether it can keep improving." Proving upper bounds is the deepest pleasure of theory, he explains, and that was exactly the problem: "I don't like that. I like breaking limits." He left theory for systems and never went back.

  2. Three months as a product engineer beat three research internships.

    Of his four internships before professorship, three at Microsoft Research turned into papers; one at Conviva (Hui Zhang's streaming company) meant writing a video player. Zhang told him to forget research for the summer: "I went purely as an engineer — I sold myself for three months, just as an engineer." He came back the only person in the field who knew which optimizations a real player could support and which it never could. "The internship in industry writing product code, doing engineering — that was the biggest help to my research." Academia, he warns, is a bubble: problems become important because other papers say so, while the engineers who see the real ones never write papers.

  3. ChatGPT was AI's Internet moment: the first AI product everyone wanted.

    His first reaction in January 2023 matched his colleagues': hype — one hot application, no reason to redirect a systems lab. His students kept pushing, some ready to skip a SIGCOMM (networking's top conference) deadline. What changed his mind wasn't how good ChatGPT was — it was who was using it. At the next table in a restaurant, people he guessed were insurance salesmen were discussing it: the first AI product everyone might use, and use constantly. "ChatGPT was the first time AI had that kind of Internet moment." He told his lab to finish the old papers and switch, two years before his tenure review: "If I don't get tenure, so be it — because this thing is genuinely important."

  4. Every wave ends in a data layer: CDN, Hadoop — his bet is KV Cache.

    His law of platforms: "Any application, once its scale is large enough, will need a data layer" that makes sharing and analyzing the data efficient. The web got CDNs, streaming reinvented them, and big data got Hadoop and Spark. His prediction: the model's internal memory becomes a more important data type than human-readable data, as long as transformers stay dominant — likely, he thinks, given industry inertia. "The way we see it, the KV cache is gold — it is the oil of the future."

  5. The KV cache is the model's video: the format it loads instantly.

    A novel and its film adaptation carry the same story, but the film is the form a person takes in fastest; the KV cache is the form a model takes in instantly, because the model produced it — "The KV cache is really like video made for the large model to watch." Why would a networking lab pick it? The field needs data heavy enough to challenge a network, and in 2023 the candidates failed: LLM text was too small, and video generation was not ready. The KV cache was both — huge, storable, shareable between models — and, like video, literally a 3D tensor.

  6. LMCache exists because industry hadn't built the storage his papers assumed.

    The lab wrote two systems: CacheGen, which compresses the cache, and CacheBlend, which reuses it beyond shared prefixes. Then they toured industry, offering to deploy both into companies' KV Cache storage systems. The answer: "What KV Cache storage system? We don't have anything like that." Engines kept the cache in GPU memory and discarded it after each request. Rather than wait, the lab built the missing layer and open-sourced it as LMCache — "You are essentially taking an academic research group and challenging the entire industry," he says, the way Spark once outran industry's own big-data projects. The gamble may have paid off. By the time companies actually wanted such a system, LMCache was already fairly complete, so many of them now use it directly, some even resell it to their own customers.

  7. You watch the model write; you pay for it to read.

    New inference chips mostly target decoding — the token generation users sit watching. His counterpoint comes with caveats (he is not a hardware person, and calls this an impression rather than a statistic), but agent workloads point the same way: "Perceptually, 90% of the time the model is decoding ... but in fact 90% of the compute is spent on prefill — on processing the input." Agent inputs run hundreds of thousands to millions of tokens against outputs of a few thousand, and every output becomes future input. Caching attacks that side: if 90% of an agent's input is a program the model already read, storing the memory skips 90% of the computation.

  8. Decouple the cache from engine, storage, and GPU — then become the standard.

    His analogy is the Internet's narrow waist, the IP layer every application converges on, where innovation is nearly impossible — IPv6 was better and still lost, because everyone already ran IPv4. As an ecosystem grows, a few structures lock in; today the OpenAI-compatible API format is one, and "KV Cache may well be the next one — or the one after that." So LMCache decouples the cache from engine, storage hardware, and GPU (not from the model, which generates it) and stays open source, because users' first demand is the freedom to swap components. One team cannot write all the integrations, which is the point: partners co-build the layer because plugging in buys compatibility with everyone else.

  9. Rivals Nvidia and AMD funded the same bet without knowing it.

    On the recent $20M round: "Nvidia's interest in our technology had nothing to do with AMD, and AMD's interest in our technology wasn't because Nvidia was interested" — each independently concluded a best-in-class KV Cache layer is key to its hardware's maximum performance, and each keeps building in-house too. His hardest founder problem: no paper deadline, no direct competitor — racing against himself. He gives the calm about a year: "By the time everyone truly realizes how important KV Cache is, every big company will be scrambling to do it." Tensormesh must be at least their equal that day.

  10. Start the company with your students; the trust is already built.

    The company was partly forced — the students behind LMCache were graduating, and without a company the project would die. But the advisor-student structure turned out to be a founding advantage: they trust his vision, he trusts their execution, and a technical push-back reads as engineering truth, not a dodge. "If I had started the company with a friend, it would never have gone this smoothly." Professors around him who founded without their students have almost all failed (he concedes the sample is too small to mean much). "My biggest pressure is not to let them waste this opportunity" — they could all be drawing very high salaries elsewhere.