The Engineering Unlocks Behind DeepSeek | YC Decoded | Highlights and Annotations by Gistr.

DeepSeek's open-source reasoning model, R1, achieved performance comparable to OpenAI's models at a fraction of the cost, causing market disruption. R1 builds upon DeepSeek's V3 base model, leveraging algorithmic innovations (detailed in prior publications) for efficiency in training and inference. These innovations include 8-bit training, a mixture-of-experts architecture, and multi-token prediction. While R1's performance is impressive, the hype also stems from its accessibility and misconceptions about its training cost. DeepSeek's work demonstrates the potential for cost reduction in AI development. Here's a breakdown of DeepSeek R1's benchmark results compared to OpenAI's models, and the significance of its reinforcement learning approach: Benchmark Results: DeepSeek R1 achieved comparable performance to OpenAI's 01 model on certain math and coding benchmarks.( ) However, just two weeks after R1's release, OpenAI released 03 Mei, which outperformed both R1 and 01 on key benchmarks. This highlights the rapid pace of innovation in the field.( ) Significance of Reinforcement Learning Approach: Emergent Reasoning: DeepSeek used a novel reinforcement learning (RL) technique called group relative policy optimization (GRPO). Through this process, the model showed emergent reasoning abilities over thousands of RL steps.( ) Learning Skills: The model learned skills like extended Chain of Thought and even experienced "aha" moments, recognizing and correcting its own mistakes.( ) Pure RL Success: R1-0, one of DeepSeek's models, was among the first large models to achieve top-tier results purely through reinforcement learning.( ) Addressing Readability Issues: While pure RL can lead to models with poor readability (e.g., randomly switching between languages), DeepSeek addressed this by introducing a "cold start" phase. They fine-tuned the model on structured reasoning examples before applying RL. This eliminated language mixing and improved output comprehensibility.( ) Reproducibility: A UC Berkeley lab reproduced DeepSeek R1's key techniques to create complex reasoning in a smaller model for a low cost ($30), demonstrating the reproducibility of the approach.( ) DeepSeek's R1: A New Era in Reasoning AI DeepSeek R1 and V3: What's the Difference? DeepSeek V3: Efficiency Innovations DeepSeek R1: Reasoning Through Reinforcement Learning Why the Hype Around DeepSeek R1? DeepSeek's Impact on the AI Landscape The announcement of DeepSeek's R1 model, achieving comparable performance to OpenAI's models at a fraction of the cost, caused significant social media panic and stock market volatility, notably impacting Nvidia's market capitalization. This highlights the disruptive potential of open-source AI models and the competitive landscape of the AI industry. DeepSeek V3, a general-purpose large language model released in December, achieved performance comparable to leading models like GPT-4. R1, released later, builds upon V3, incorporating algorithmic improvements to optimize reasoning abilities, resulting in comparable performance to OpenAI's 01 and Google's Flash2 on specific benchmarks. DeepSeek V3 utilizes a mixture of experts architecture, activating only 37 billion of its 671 billion parameters for each token prediction, significantly reducing computation compared to models without this architecture. This, combined with novel training techniques, stabilizes performance and increases GPU utilization.V3 employs multi-head latent attention (MLA) to compress key and value matrices, reducing KV cache size and boosting throughput. Furthermore, multi-token prediction (MTP) allows V3 to anticipate multiple future tokens, improving data efficiency and generating smoother, more coherent outputs. Due to hardware constraints and US export controls on GPUs to China, DeepSeek focused on maximizing the efficiency of their existing GPU cluster. Nvidia's integrated solution, including networking, software, and developer experience, provides a significant advantage in maximizing GPU utilization, unlike the typical 35% utilization seen in FP8 training. DeepSeek optimized V3 for efficiency by using 8-bit floating-point format instead of the usual 16 or 32-bit, achieving significant memory savings without performance loss. This, along with an FP8 accumulation fix preventing compounding numerical errors, enabled more efficient training across thousands of GPUs, reducing costs while maintaining model quality. The hype surrounding R1 stems from its accessibility, allowing free download and customization, and its near state-of-the-art performance at a fraction of the cost of other models. However, the actual cost of training V3 was significantly lower than initially reported, and the work is reproducible, demonstrating that cost-effective innovation is still possible in the AI field. While most LLMs benefit from step-by-step prompting, reasoning models like R1 are specifically trained to break down complex problems. DeepSeek used reinforcement learning, particularly a novel technique called Group Relative Policy Optimization (GRPO), to train R1, achieving top-tier results purely through reinforcement learning without human or AI feedback examples.R1 demonstrated impressive reasoning skills, including extended chain of thought and self-correction, achieving comparable performance to OpenAI's 01 on certain benchmarks. This success showcases the potential of pure reinforcement learning in training advanced reasoning models, though the model's outputs initially suffered from language mixing issues. Accessibility : DeepSeek's models are freely accessible through their website and app, allowing for customization and local execution. Cost Efficiency : DeepSeek's efficiency improvements enable near state-of-the-art performance at a fraction of the cost compared to other reasoning models. Misconceptions about Training Costs : The $5.5 million training cost for V3 refers only to the final training run and does not include the cost of R1 training, R&D, or hardware expenses. Reinforcement Learning (RL) : DeepSeek uses RL to shape the model's behavior based on feedback and reward signals. RL for Reasoning : R1 applies RL specifically to train the model to think step-by-step through complex problems. Training Pipeline : R1 is trained on problems with verifiable outputs, and the model is rewarded for outputting correct answers. Simple Evaluation : DeepSeek uses simple rules to evaluate the model's final output based on accuracy and formatting, avoiding complex AI-based grading. Group Relative Policy Optimization (GRPO) : A novel technique used to update the model through RL, enabling the emergence of reasoning skills like extended chain of thought. Cold Start Phase : Fine-tuning on structured reasoning examples before RL to improve readability and eliminate language mixing issues. 8-bit Floating Point Format : DeepSeek V3 uses 8-bit floating point format for its calculations, compared to the usual 16-bit or 32-bit format. This significantly reduces memory usage without sacrificing performance. FP8 Accumulation Fix : Periodically merges calculations back into a higher precision FP32 accumulator to prevent small numerical errors from accumulating during training. Mixture of Experts Architecture : DeepSeek V3 has 671 billion parameters, but only 37 billion are activated for a given token prediction, saving computational resources. Multi-Head Latent Attention (MLA) : A technique introduced in DeepSeek V2, compresses key and value matrices into a latent representation, reducing KV cache size and boosting generation throughput. Multi-Token Prediction (MTP) : Predicts multiple future tokens at each step, improving data efficiency and faster learning, leading to smoother and more coherent outputs. DeepSeek V3 : A general-purpose language model released in December 2023, achieving comparable performance to other base models like GPT-4, Claude 3.5, and Gemini 1.5. DeepSeek R1 : A reasoning model built on top of DeepSeek V3, released in January 2024. It incorporates algorithmic improvements to optimize reasoning abilities, achieving comparable performance to OpenAI's GPT-1 and Google's FLAN 2.0 on complex reasoning benchmarks. New Players on the Frontier : DeepSeek demonstrates that there is still room for new players in the AI field. Optimizing the AI Stack : DeepSeek emphasizes the importance of optimizing GPU workloads, improving software at the inference layer, and developing AI-generated kernels. Reducing the Cost of Intelligence : DeepSeek's advancements contribute to the decreasing cost of AI applications, making them more accessible for both consumers and businesses.