Zhibin Xiao, an architect who helped build Oracle's database accelerators and Alibaba's first AI chip, on AI software bottlenecks, the long chip-development cycle, and why NVIDIA’s execution stands out in the chip industry.

Zhibin Xiao is founder and CEO of ZFLOW AI, which uses simulation to reduce the need for experiments on expensive AI hardware. He entered Zhejiang University's Chu Kochen College (the honors college) in 1999. A summer course with UT Austin professor Yale Patt drew him to computer architecture. His PhD at UC Davis focused on digital signal processors with many processing cores. He studied how to keep data flowing on-chip and draws parallels with today’s dataflow AI processors. From 2012 to 2017, he worked on Oracle's Software in Silicon team, building database acceleration into SPARC CPUs. He then helped build Alibaba's first AI chip, Hanguang 800, and became Chief Architect at a startup designing AI chips optimized for sparse computation before founding ZFLOW AI.

ZFLOW AI combines system simulation, model compilation, runtime scheduling, and profiling to explore how models run on existing or proposed hardware. In May 2026, it reported that a simulation-guided configuration for DeepSeek V4-Pro on eight NVIDIA B300 GPUs achieved 1.54 times the peak throughput of monolithic serving, with a two- to threefold improvement in tail latency at high concurrency. The configuration separated prefill from decode, and the results were measured on a neocloud provider's hardware. Xiao's larger ambition is a vendor-neutral simulation and optimization platform for chip designers and AI infrastructure teams.

In our in-depth conversation, we discuss:

  1. Xiao’s journey from video-codec processors to Oracle’s database accelerators, Alibaba’s Hanguang 800, and AI system optimization
  2. Why Xiao founded a software-focused system optimization company after 20 years designing chips
  3. Why Taalas’s dedicated chips can be cost-effective when a model’s task stops changing
  4. The CPU projects that could take three to five years, and why verification teams outnumbered designers
  5. Why Xiao expects the next great chip company to organize its entire workflow around AI
  6. What process-node numbers describe, why power savings can exceed speed gains, and why on-chip memory complicates the comparison
  7. How a prompt becomes tokens: scheduling, prefill, KV cache, and decoding's data-movement problem
  8. How Groq, Cerebras, and SambaNova address the memory-bandwidth bottleneck in token-by-token decoding, compared to GPUs
  9. NVIDIA's hardware advantages in Xiao's view: execution speed and access to high-bandwidth memory and foundry capacity
  10. Why chip engineers often earn less than software engineers, and how AI could reshape pay across the tech industry

My biggest takeaways from this conversation:

  1. Xiao believes the main AI bottleneck has shifted to systems and software, so his new company ZFLOW AI starts with optimization.

    As Moore's Law slows, improving individual chips becomes harder, and many companies already compete to do it. Xiao sees a different problem to solve: "The AI chip is no longer the main bottleneck. It's the system and software." ZFLOW AI uses simulation to find promising configurations, then calibrates its predictions with measurements from real hardware. It brings that process together with model compilation and runtime scheduling. Xiao is also exploring partnerships with open-source serving frameworks so engineers can begin tuning before hardware arrives. The commercial challenge remains: an optimization business needs customers who trust and need its work, or access to resources such as GPU capacity.

  2. As tasks and models stabilize, moving a model into a custom chip can offer better cost efficiency.

    Xiao remains open to designing chips again if simulation reveals a promising opportunity. He gives examples such as robot vacuums and parking-lot monitoring, where a model may already meet the needs of a task that changes little. He sees that potential in Taalas, which builds models directly into chips: "For a stable, specific task, hardwiring a model into silicon can make economic sense." The design still allows some changes, he explains. Certain metal layers can be modified and the revised design sent for fabrication, at a lower cost than a new tape-out using a full set of masks. That flexibility still depends on manufacturing, so changing the model is a very different undertaking from updating software.

  3. Newer chip processes can improve power efficiency more than speed, while SRAM may not shrink.

    Xiao explains that labels like 7 nm and 3 nm now identify generations of chip manufacturing technology. They no longer refer to a single physical dimension. For TSMC's move from N3 to N2, he estimates power savings of 10–15 percent and speed gains of only 3–5 percent. Less power consumption at the same performance means less heat, allowing more compute in a rack. But logic circuits and on-chip memory do not necessarily shrink together. In his rough example, half of an AI chip is logic and half is SRAM, a type of on-chip memory whose scaling has slowed across the 7-to-3 nm generations. "Going to a smaller process no longer necessarily shrinks the SRAM cell." He says designs that rely heavily on SRAM, such as Groq's and Cerebras's, may need to change that balance or explore 3D stacking.

  4. Software can narrow the performance gap between 7 nm and 2 nm systems, but both can still be optimized.

    Asked to compare systems using 7 nm and 2 nm chips, Xiao puts the underlying performance gap at roughly two to three times, taking power consumption, compute density, and interconnect speed into account. He estimates that software improvements could deliver gains of a similar size. These include separating input processing from token generation, using lower-precision numbers, speculative decoding, multi-token prediction, and better scheduling. He also estimates that optimization could raise chip utilization from an industry average of 30–50 percent to 80–90 percent. His caveat is essential: "An optimized older chip can close the gap with a newer chip that's poorly optimized." A system using newer chips can apply the same software improvements, so optimization does not guarantee that older hardware will catch up. More detailed work on hardware and software together also requires more engineering time and resources.

  5. Dataflow chips can reduce memory traffic during decoding and work alongside GPUs.

    During prefill, a model processes its input in parallel, making good use of a GPU's computing capacity. During decoding, it generates one token at a time, repeatedly accessing model weights and the KV cache, which stores information from earlier tokens. Xiao explains that moving this data can become the bottleneck, leaving GPU computing resources underused. Groq uses software scheduling to keep data moving among cores on the chip, reducing trips to external memory. Cerebras applies a similar approach across an entire wafer, while SambaNova combines dataflow with external memory. There is still a capacity limit: models that no longer fit on Groq or Cerebras hardware require more chips. Xiao sees these architectures as useful partners for GPUs because different designs suit different parts of the workload.

  6. NVIDIA's hardware advantage also includes strong R&D execution and a secure supply chain.

    Xiao sees NVIDIA's development speed as a major strength: "From a chip designer's perspective, its execution is the best in the industry." He points to support for sparse computation, which NVIDIA put into GPUs while other companies were still writing papers. Annual product releases are possible because development teams work in parallel. He estimates that products such as B300 and GB300 began development two or three years before release. A competitor may design an effective inference chip yet still struggle to obtain high-bandwidth memory (HBM) or manufacturing capacity at TSMC. Xiao expects NVIDIA's advantages in development and the supply chain to continue as long as Jensen Huang leads the company.

  7. Chip verification helps catch design errors before manufacturing begins.

    Xiao recalls CPU projects that took about two years to design and three to five years to deploy, including manufacturing, system integration, and software. Architects first set the requirements and balance power, performance, area, and cost. Microarchitects turn those requirements into a concrete hardware design. Before AI, he says, a chip company might have two or three verification engineers for every design engineer, using simulation and emulation to check for errors. These checks happen before tape-out, when the design is sent to a foundry for manufacturing. "A failed tape-out can cost hundreds of millions of dollars, not to mention a year or two." In the process he describes, wafers arrive about three to four months later. Testing, cutting the wafer into dies, packaging, board-level validation, and customer samples all come before mass production.

  8. Xiao expects the next great chip company to organize its entire development process around AI.

    Xiao sees verification as a natural starting point for AI agents because they have inputs, outputs, and simulators to check their results. Synopsys and Cadence already use AI in electronic design automation tools for physical design, including placement and routing. But tools for deciding what chip to build and defining its architecture remain scarce, so human judgment is still central. Xiao also sees an organizational obstacle: established companies have familiar workflows, and engineers may resist handing their work to AI. The company he imagines would organize architecture, design, verification, simulation, and physical implementation around AI from the outset.

  9. Power plants and production lines can't be expanded overnight, so software must make the most of existing compute capacity first.

    Asked what most constrains inference, Xiao first names compute capacity, then emphasizes the ability to keep lowering the cost of each token. Models change quickly, hardware varies, and teams often lack time to tune every deployment, leaving GPUs and their features underused. "Cost pressures and physical constraints force them to become more efficient, and to automate that optimization." ZFLOW AI aims to turn its team's chip and systems knowledge into tools that agents can use to keep optimizing. Software can improve the use of today's resources, while physical expansion follows a slower schedule. In the US, he notes, building a power plant or expanding the grid can take five years or more. Foundry construction and the number of lithography machines delivered each year also limit how quickly capacity can grow.

  10. As cloud companies invest more in hardware, Xiao expects top pay to remain concentrated among a few specialists.

    Xiao says chip engineers in Silicon Valley have often earned less than software engineers, and he attributes the difference to company spending. Chip businesses pay for fabrication, design-tool licenses, and chip IP, so salaries account for only part of their costs. Internet companies historically invested more in people, then served more customers at relatively little additional cost once the software was built. He sees AI changing that pattern: "Cloud companies are starting to resemble semiconductor companies: capital spending dominates." As infrastructure takes a larger share of spending and AI takes over some tasks, he expects many software engineers to work primarily with AI tools. In his forecast, people with scarce expertise will continue to command exceptional pay. He adds that even today's widely reported AI salaries go to a very small group.

  11. Xiao chooses jobs that let him understand the whole system and learn something new.

    Oracle's new accelerator team gave Xiao a chance to work on CPU design and database software together. He was responsible for its compression architecture. The accelerator used a preloaded codeword table to decompress data before processing database queries. If the database changed, the team could scan it again and generate a new table. At Alibaba, his responsibilities included both architecture and product management, from assessing use cases and return on investment to choosing chip IP and working with foundries. He recommends innovative projects with a broad scope when personal circumstances allow, while recognizing stability and immigration needs as valid reasons to choose a more established role. Becoming CEO brought another challenge: "Selling software really is different from selling chips." He now works with customers and partners to decide how to price the product and whether to sell consulting, services, or subscriptions.