MiniMax Introduces Groundbreaking Open Source LLM with Extended Context Capabilities

Jimmy Jing

— 2 min read
MiniMax Introduces Groundbreaking Open Source LLM with Extended Context Capabilities

MiniMax, a prominent Singaporean technology company known for its generative AI video model Hailuo, has recently made a significant announcement. The company has released and open-sourced a new family of models known as the MiniMax-01 series, which promises to redefine long-context AI applications.

A Massive Context Window

The new series includes the MiniMax-Text-01, a foundation large language model, and MiniMax-VL-01, a visual multi-modal model. Notably, MiniMax-Text-01 boasts a remarkable capacity for handling up to 4 million tokens in its context window. This advancement supersedes Google's Gemini 1.5 Pro, which supports a 2 million token context window.

The extended context window enables handling vast amounts of data equivalent to a small library in a single input/output exchange. This capability is expected to enhance AI agent development as demand for complex context understanding grows.

"MiniMax-01 efficiently processes up to 4M tokens – 20 to 32 times the capacity of other leading models," the company announced. "This positions MiniMax-01 to support forthcoming applications that demand extensive context handling."

Current downloads are available via Hugging Face and GitHub, with direct use possible on Hailuo AI Chat and through MiniMax's API.

Competitive Pricing and Scalability

MiniMax presents its text and multi-modal APIs at very competitive rates:

  • $0.2 per 1 million input tokens
  • $1.1 per 1 million output tokens

For context, OpenAI's equivalent GPT-4 service costs $2.50 per 1 million input tokens. This cost efficiency is complemented by the incorporation of a Mixture of Experts (MoE) framework with 32 experts to optimize both memory and computational resources.

Introducing Lightning Attention Architecture

The innovative Lightning Attention mechanism underpins MiniMax-01, moving away from the traditional Transformer architecture by significantly lowering computational complexity. The model utilizes 456 billion parameters, with 45.9 billion activated per AI inference. By integrating linear layers with SoftMax, MiniMax achieves near-linear scaling for lengthy inputs.

The architecture is further optimized by rebuilding training frameworks, introducing:

  • MoE All-to-All Communication Optimization: Reduces GPU communication overhead.
  • Varlen Ring Attention: Maximizes efficiency for long-sequence processing.
  • Efficient Kernel Implementations: Enhances performance.

Performance Benchmarks

MiniMax-01 exhibits performance comparable to top-tier models such as GPT-4 and Claude-3.5, especially excelling in long-context tasks. The MiniMax-Text-01 has achieved a perfect score in specific evaluation tasks and demonstrates minimal performance loss with increased input length.

The company is committed to continuous improvements, aiming to expand the model's capabilities further. Open-sourcing the MiniMax-01 is seen as a strategic move to foster foundational AI developments in anticipation of future AI agent demands.

Collaboration and Future prospects

MiniMax welcomes developers and researchers to engage with the MiniMax-01 series, inviting technical collaboration and feedback. With their efforts to provide cost-effective, scalable AI, MiniMax positions itself as an influential player in the evolving AI landscape. This new series offers unprecedented opportunities for developers to leverage long-context AI for future innovations.

Read more