SambaNova Systems wins Coolest Technology Award at VentureBeat Transform 2024

Kunle Olukotun accepts Coolest Technology Award at VB Transform 2024

Palo Alto-based AI chip startup SambaNova Systems was awarded “Coolest Technology” at VentureBeat Transform 2024 in San Francisco. The company, founded in 2017, has raised $1.1 billion to date and is currently valued at $5 billion.

We spoke with Kunle Olukotun, co-founder and chief technologist of SambaNova, about the company’s technology and recent achievements. Olukotun is also the Cadence Design Professor of Electrical Engineering and Computer Science at Stanford University. “I’m most well known for being one of the pioneers of an idea called multicore,” he said. “We designed a multiprocessor for the data center.” His previous venture, Afara Web Systems, commercialized multicore technology before being acquired by Sun Microsystems 2002.

SambaNova, founded in 2017, emerged from a confluence of trends. Olukotun elaborated, “We saw the opportunity to combine software insights about [ML] models with efficient hardware implementations.” The company’s genesis was rooted in the realization that machine learning was becoming a new paradigm for computing.

The company’s approach differs from traditional GPU manufacturers. “Unlike NVIDIA GPUs, which were repurposed graphics and gaming chips, this is designed expressly for AI computation,” Olukotun stated. He added, “It focuses on optimizing data movement, which is the critical bottleneck to the performance of high performance inference.”

The company’s latest chip, the SN40L, showcases their innovation. Unlike traditional GPUs repurposed for AI, SambaNova’s chip is built from the ground up for AI computation. Olukotun explained their unique “reconfigurable dataflow” architecture: “Our innovation was to say, look. If that’s the way that all these models are expressed, why don’t we come up with hardware where that’s the native execution mode?”

This approach allows SambaNova to “streamline data movement and provide the lowest latency inference, the highest number of concurrent LLMs at the same time, and the lowest switching time between different LLMs,” according to Olukotun. The chip’s design focuses on optimizing data movement, which Olukotun identified as “the critical bottleneck to the performance of high-performance inference.”

Customer traction

SambaNova Systems has been gaining significant traction with high-profile customers across various sectors. 

In the financial services industry, OTP Group, the leading banking group in Central and Eastern Europe, partnered with SambaNova to build Europe’s fastest AI supercomputer. This multi-rack AI system, powered by SambaNova’s Dataflow-as-a-Service GPT, aims to transform OTP’s services for its 17 million customers across 11 countries. 

In the public sector and research domain, SambaNova has made substantial inroads. The company has expanded its collaboration with Lawrence Livermore National Laboratory (LLNL), integrating its spatial data flow accelerator into LLNL’s Computing Center to enhance cognitive simulation programs. 

Similarly, Los Alamos National Laboratory has chosen to scale up its existing deployment of SambaNova systems to advance its generative AI and LLM capabilities. This partnership aims to support the lab’s mission objectives in national security, science, energy and environmental management. 

In the energy sector, Saudi Aramco, the world’s largest energy company, has deployed SambaNova’s hardware to power its internal LLM called Metabrain, which has grown from a few hundred users to thousands within months. 

How SambaNova stacks up

In the competitive AI chip market, SambaNova Systems faces challenges from industry giants, cloud providers, and several other high profile startups. 

Nvidia currently dominates the market with its H100 GPU, backed by a robust ecosystem. Cloud giants like Google, Amazon and Microsoft are developing their own AI chips, optimized for their specific environments. Among startups, Cerebras Systems’ WSE-3 and Groq’s Tensor Streaming Processor offer formidable competition. 

SambaNova’s approach centers on optimizing data movement, which Olukotun identifies as “the critical bottleneck to the performance of high-performance inference.” The solution also differentiates itself through its focus on handling multiple LLMs concurrently and efficiently switching between them. “You can’t get this capability of having high speed, large numbers of models, the ability to switch between models instantly, on any other platform,” Olukotun emphasized. 

SambaNova Systems recently set a new record for LLM inference performance, as independently benchmarked by Artificial Analysis. Their Samba-1 Turbo achieved 1,084 output tokens per second on Meta’s Llama 3 Instruct (8B) model, more than eight times faster than the median output speed across providers. 

This performance was achieved using just 16 chips, while maintaining full precision and the ability to concurrently host up to 1,000 Llama3 checkpoints on a single 16-socket SN40L node. This breakthrough, attributed to SambaNova’s Dataflow architecture, offers significant advantages in speed, efficiency and cost-effectiveness compared to GPU-based competitors. The company claims this performance translates to a 10X lower total cost of ownership, positioning SambaNova as a leader in high-speed, high-capacity AI inference solutions for enterprises.

The battle for developers

While SambaNova’s technological innovations are impressive, the company’s success will depend not just on the performance of its chips, but on its ability to build a thriving developer ecosystem.

SambaNova is taking multiple steps to court developers. They recently introduced their new Fast API, which provides access to pre-trained models and chip capabilities, including “full precision, 8 billion and 70 billion Llama 3 models” running on the SN40L. This move aims to lower the barrier to entry for developers and startups looking to leverage SambaNova’s technology.

In addition, SambaNova has announced SambaVerse, a unique playground and API for developers. SambaVerse allows developers to test and compare hundreds of available open-source LLMs from a single endpoint. This tool enables developers to directly compare model responses for any given application, querying multiple models simultaneously and displaying results in real-time.

Olukotun sees the next generation of AI applications as focused on agentic AI, which refers to AI systems that can act autonomously to achieve goals, and often involves decision-making and interaction with their environment. 

SambaNova’s architecture could enable more dynamic and responsive agentic AI systems, potentially allowing for real-time adaptation to changing tasks or environments. This capability is important because agentic AI systems often need to leverage different models for various tasks or decision-making processes. 

For instance, an agentic AI might use one model for natural language understanding, another for reasoning, and yet another for generating responses or actions. Smaller models might be used for quick, specialized tasks, while larger models could handle more complex reasoning or generation tasks. 

Through both hardware capabilities and developer tools like Fast API and SambaVerse, developers using SambaNova’s platform could gain a significant advantage in building sophisticated agentic AI systems.

The future

SambaNova’s success will hinge on its ability to deliver superior performance and efficiency for agentic AI applications while cultivating a rich developer ecosystem. The introduction of Fast API and SambaVerse demonstrates SambaNova’s commitment to providing developers with the tools needed to build and experiment with advanced AI systems. 

SambaNova’s architecture, optimized for running and rapidly switching between multiple LLMs, aligns well with the complex, dynamic processing needs of agentic AI systems. As these applications become more sophisticated, requiring seamless integration of various specialized models, SambaNova’s approach may become increasingly valuable.

Congratulations to SambaNova Systems on winning the “Coolest Technology” award at VentureBeat Transform 2024! This recognition underscores the company’s innovative approach to AI computation and its potential to reshape the enterprise AI landscape. 

For those interested in learning more about SambaNova’s technology, including their SN40L chip, Samba-1 model, and developer tools, visit their website at sambanova.ai. Developers can also explore SambaVerse at sambaverse.sambanova.ai to experience firsthand how SambaNova’s platform can enhance AI development workflows.