Editor’s note. This is the first post in China AI Focus, our deep-dive column on Chinese AI from primary sources. The deal here is simple: every fact links to first-party evidence, every judgment is labeled as ours, and every viral claim gets checked against the primary source before we repeat it. We normally publish midweek; this one ships today because DeepSeek published the announcement this morning (October 17, 2026), and one detail in it has a hard deadline attached.

What happened, verified

Everything in this section comes from DeepSeek’s own release notes and model card — first-party, published October 17, 2026.

1. The model. DeepSeek updated its Flash line to V4.1-Flash-1017, live simultaneously across web, app, and API. The “Flash” name does not mean small: the model card lists 763B total parameters in mixed precision and a 1M-token context window (max output ≥256K), released under an MIT license. The company reports shorter task runtimes and shorter outputs versus the previous Flash — cost falls twice: better performance per token, and fewer tokens per task. The release claims a “new performance–cost Pareto optimum” across several agent harnesses, with specific tuning for DeepSeek Harness, Tencent’s CodeBuddy , and WorkBuddy. DeepSeek recommends upgrading to DeepSeek Harness 0.2.1-rc.1 or later.

2. The benchmark table — company-reported, and more interesting than the hype. DeepSeek’s own model card scores the new Flash at 77.0 on DeepSWE v1.1 (prior Flash: 74.2) and 42.3 on Terminal-Bench 4.0 (prior: 31.2). The same table puts it ahead of GLM-5.3 (66.9) and GPT-5.6 Sol (73.0) on DeepSWE — and ahead of Opus 5 (74.0) there too. But on Terminal-Bench 4.0, Opus 5 still leads 51.8 to 42.3; on ProgramBench, 37.0 to 29.1; on NL2Repo-Bench, 75.3 to 68.2. Keep that asymmetry in mind when you see the community shorthand “on par with Opus 5” — it is true on the agentic-SWE benchmark DeepSeek optimizes for, and not yet true on the others. All numbers above are company-reported until independent evals land.

3. The training story. The announcement credits “reinforcement learning based on self-feedback” for the jump in whole-project generation — what DeepSeek’s own headline calls “wish-style programming” : describe the thing, get the thing. The demos are pointed: a 3D model of the Forbidden City, a browser racing game, a Portal 2 remake built in the Godot engine, and “Valley of Ages,” a 3D scene of historical development.

4. The search API — the quieter, bigger move. DeepSeek opened its self-built Search and Fetch APIs, each priced at RMB 5 per 1,000 calls (approx. US$0.70). The underlying index claims hundred-billion-scale coverage of global web pages, multilingual retrieval, and — notably — filtering of SEO gaming, GEO (generative-engine-optimization) manipulation, and low-quality AIGC content. Today, Search and Fetch invoked as server-side tools from DeepSeek Harness, Claude Code, or Codex are free. That ends at 00:00 Beijing time on November 1, 2026, when calls start billing per thousand.

5. Open weights, same day. V4.1-Flash-1017 is downloadable now from Hugging Face and ModelScope under MIT — no wait, no gate, no delay behind the API, and no strings on commercial use.

Five hours in: the reaction

We treat social platforms as heat measurement, not as evidence of facts — so read this section as propagation data and named individual opinions, nothing more. Within about five hours of the announcement, DeepSeek’s own launch post had drawn roughly 20K likes, 2.4K reposts, and 2.2M views, and the topic was sitting in X’s trending feed.

Three threads in the reaction are worth your attention:

• The comparison chatter is already benchmark-specific. Early community assessments — e.g. from @kimmonismus and @imjustnewatai — landed on “roughly on par with Opus 5,” citing the model card’s table, with the asymmetry we detail in section 2. Some users also circulated an off-peak input price of about US$0.15 per million tokens; the model card doesn’t state pricing, so verify on DeepSeek’s pricing page before you budget around it.

• The deployment joke making the rounds this morning: running 763B locally on two NVIDIA DGX Spark units. The subtext is real — an MIT-licensed 763B model at this quality tier gives private deployment a credible option, which is exactly the pressure point on Western API pricing.

• The memes are about generation, not chat. The viral test of the morning came from @bijanbowen, who asked the model for a “model slug” and got a 3D model of an actual garden slug. A perfect literal-minded fail — and also the point: people are stress-testing whole-artifact generation (3D, games, repos), which is precisely the capability DeepSeek claims the self-feedback RL bought.

The model is the headline. The search API is the story.

A new Flash checkpoint would be routine — DeepSeek ships fast and often. Three things make today unusual.

First, DeepSeek is treating the harness as the product surface. The announcement doesn’t just name a model; it names three harnesses it tuned for, recommends a specific harness release version, and publishes its performance-per-cost measurement setup. When a lab co-optimizes model, harness, and eval methodology together, it is selling an agentic stack, not a chat model. That is the same playbook that made Claude Code and Codex sticky — now arriving from the price-cut side, with an MIT license as distribution.

Second, DeepSeek just posted a price floor for the search-for-agents layer. Every serious agent needs retrieval, and search has been the quiet line item in agent economics — usually a third-party dependency. At RMB 5 per 1,000 calls, DeepSeek is pricing retrieval like a utility. The anti-GEO, anti-AIGC-slop filtering claim is the part worth tracking: if the index genuinely surfaces primary sources over generated filler, that is a retrieval-quality edge, not just a price cut. If it doesn’t hold up, this is a land grab with a teaser rate. Either way, everyone pricing agent infrastructure now has a number to beat or explain.

Third — read this line again — DeepSeek infrastructure runs inside Claude Code and Codex as server-side tools. Chinese model infrastructure is already embedded in the default toolchains of Western developers, and from November 1 those developers will be paying DeepSeek, in RMB-denominated pricing, for the search layer inside them. However you feel about that, it is a fact about the market’s direction, not a prediction.

Can you use it outside China?

• Open weights: yes, MIT. Hugging Face globally, ModelScope in China. Commercial use permitted by the license; run it wherever your compliance posture allows.

• API: yes, via DeepSeek’s platform. Web, app, and API switched simultaneously; the announcement doesn’t restrict by geography. Standard caveat: check the platform’s current terms for your jurisdiction before production use.

• Search/Fetch API: same platform, same terms — and remember the November 1 billing switch if you’re calling it through a harness.

What to do before November 1

1. Audit your harness configs. If you use DeepSeek Harness, upgrade to 0.2.1-rc.1 or later. If you run Claude Code or Codex with server-side tools, check whether Search/Fetch calls route through DeepSeek — after November 1 they bill at RMB 5 per 1,000.

2. Estimate the bill. Count your agent’s search calls per task and multiply: at RMB 5 per 1,000 calls, a million search calls run about RMB 5,000 (approx. US$700). Decide whether the free period is worth a load test now, before the meter starts.

3. Re-run the comparison on your own workload. The card’s table says Opus 5 still wins terminal and program-synthesis tasks; DeepSWE-style agentic work favors the new Flash at DeepSeek’s published price point. Route accordingly — and treat all of it as company-reported until your own eval says otherwise.

4. Benchmark the search layer against your current provider on your real query mix — especially if you operate in a vertical where primary-source retrieval matters more than volume.

5. Smoke-test the weights. A 763B mixed-precision release has real hardware implications; measure before you commit a lane to it.

What would change our read

• Independent evals fail to reproduce the DeepSWE and Terminal-Bench gains, or the gains only appear inside DeepSeek’s own harness.

• The anti-GEO/anti-AIGC filter degrades retrieval in practice — over-blocking fresh content or missing non-English primary sources.

• Pricing or terms shift after November 1, or API availability tightens outside China.

• The community’s “on par with Opus 5” framing collapses on independent multi-benchmark runs.

Sources and method

• Primary source (Chinese, first-party): DeepSeek API docs news, October 17, 2026 — the release facts trace here.

• Primary source (model card, first-party): Hugging Face — deepseek-ai/DeepSeek-V4.1-Flash-1017 — license, parameters, context length, and the benchmark table trace here. Also on ModelScope.

• Propagation data: X trending feed and public post metrics, observed October 17, 2026 — used only as heat evidence and for attributed individual opinions, never as fact sources.

• aigc.news event

Facts and judgments are separated by section above; social claims are named as such. Corrections welcome — reply to this email and we’ll log the change.

AIGC.NEWS — China AI, from the inside. Verified from primary Chinese sources.

October 5–11, 2026 · China AI #1

AIGC Weekly now focuses on China AI. Each issue connects the week’s important developments with their implications outside China. Earlier issues, through #92, remain in the archive.

Our read this week: capital is expanding the ambitions of Chinese AI companies; product control and overseas deployment will show how much of that ambition reaches users.

This week at a glance

• Capital: DeepSeek’s reported mega-round; Manus raises fresh funding after its separation from Meta.

• Products and architecture: Vidu adds reference controls; DiffuSpace funds a diffusion-language-model challenger.

• Overseas adoption: Ecosia considers open models through European infrastructure.

1. DeepSeek: a second external round in four months

Capital · October 6 · Reported; closing pending

Reuters reports that DeepSeek is set to raise more than RMB 80 billion—about $11.93 billion. Sources expect an October close; DeepSeek had not commented. The report follows its approximately $7.4 billion first external raise in June and says commitments exceed the new round’s original RMB 50 billion target.

The change is the pace of external financing: a second large round within four months would give DeepSeek a much larger capital base. For competing labs, its funding capacity becomes another factor alongside model quality and API prices. For infrastructure suppliers, an expanded budget is a potential opportunity, but orders and capacity commitments still need evidence.

Read more: DeepSeek’s funding push →

2. Vidu Q4 Preview: reference controls for recurring characters

Video · October 7 · Preview

ShengShu’s Vidu Q4 Preview supports up to 15 image references and three audio references, with advertised 2K/4K output and launch pricing starting at $0.014 per generated second.

The practical question is whether a creator can carry the same character, wardrobe, product and voice across several shots. Reference controls give studios more explicit inputs for those choices. If they reduce rejected takes, they could improve production economics even without a lower headline price. The starting rate alone does not establish the cost of an accepted scene.

Try this: make three shots with the same character and voice. Track continuity, retries and total spend per usable shot. Confirm the preview’s settings and plan before comparing costs.

Read more: Vidu Q4 Preview →

3. Manus: fresh capital after the Meta unwind

Agents · October 8 · Company-announced financing

Manus’s parent, Butterfly Effect, said it completed a round exceeding $500 million, co-led by Boyu Capital and IDG Capital. Reuters and Shanghai Securities News covered the announcement. It follows Manus’s return to independent operations after Beijing ordered Meta to unwind its acquisition.

That sequence matters for overseas buyers and competitors: Manus’s ownership path has changed, while funding allows its standalone agent business to continue developing. The acquisition unwind shows why a China-origin agent company’s corporate structure deserves attention alongside its product capabilities.

For teams evaluating Manus, the useful diligence now includes which entity contracts with customers, the current data terms and how the service handles ongoing projects after the separation.

Read more: Manus’s financing and independent operations →

4. DiffuSpace: industrial capital backs a different generation order

Models · October 9 · Financing reported by independent outlets

ChinaVenture and STAR Market Daily report two completed DiffuSpace funding rounds, with Huawei’s Hubble and Horizon Robotics among the investors. STAR Market Daily puts the combined amount near RMB 500 million, citing people familiar with the company, and reports plans for a larger model release with open sourcing in October.

Diffusion language models can fill or revise multiple text positions through iterative updates. That offers a different approach to constrained editing and generation latency; any speed gain must be evaluated together with quality and the number of refinement steps.

Industrial investors make model-and-hardware fit worth watching, particularly for low-latency applications. Their participation does not establish a deployment contract. The team’s earlier Dream 7B is a 2025 release; this week’s news funds the next stage of development.

Read more: DiffuSpace’s diffusion-model bet →

5. Ecosia: Chinese models through European infrastructure

Overseas adoption · October 9 · Founder interview

Ecosia founder Christian Kroll explained its move away from Mistral in terms of affordable, efficient open models and said the company would probably use Melious.ai. The provider’s catalog includes Qwen, GLM and Kimi; it describes European hosting. The interview does not establish the model used for every response or confirm a completed migration of the whole service.

For Chinese labs, this points to a distribution route beyond their own APIs: overseas operators can host model weights and maintain the local customer relationship. For European buyers, it makes the model developer and inference provider separate procurement choices. Licensing, routing and data processing therefore need to be checked at both levels.

Read more: Ecosia’s model choices →

Briefly: Moonshot’s IPO watch

Bloomberg reports Moonshot at a $50 billion valuation, targeting a Hong Kong IPO in early 2027; company confirmation remains pending.

Read more: Moonshot’s valuation and IPO plans →

What to watch

• DeepSeek: Will the reported $12B round close this month, and on what terms?

• DiffuSpace: Will its next model arrive in October, and how will it balance speed and quality?

• Ecosia: Which Chinese models will it use, and where will they run?

Subscribe now

After the release, OpenClaw quickly added support for it. From the pricing structure, Flash is actually extremely cheap, while Pro is relatively expensive. The gap between the two is around 12x.

V4 also quickly appeared on OpenRouter’s usage ranking, which shows that people were already trying it in real workflows.

Then, on the second day, DeepSeek launched a 75% discount for V4-Pro API.

The DeepSeek-V4-Pro API is now available at a limited-time 75% discount, ending at 15:59 UTC on May 5, 2026.

My guess is that the initial pricing of Pro was indeed a bit too high. For many people doing coding or agentic coding, they would naturally use Pro or Pro Max-like settings. After trying it for a day, it could easily cost dozens of RMB. Compared with a fixed coding plan, that is simply too expensive.

DeepSeek may have noticed this quickly, so it introduced the limited-time discount.

This also puts OpenRouter in an awkward position. If OpenRouter cannot adjust prices in time, many users will simply go back to the official API.

Then, on the third day, DeepSeek announced another price cut:

Starting immediately, the input cache hit price for all DeepSeek API products has been reduced to one-tenth of the original price. Build applications more efficiently at a lower cost.

This cache price cut is the real killer move.

Once the cache price drops to one-tenth of the original level, few vendors can really follow. DeepSeek is clearly starting to grab market share. My feeling is that this price may actually be close to the normal price level after Huawei’s inference servers begin shipping at scale.

It almost feels intentional.

First, set the price high.
Then cut it twice.
Get trending twice.
And finally land at a price point that looks extremely competitive.

The result is that DeepSeek-V4 suddenly looks like one of the highest cost-performance options on the market.

This is not just a model release.

It is a pricing campaign, a developer acquisition campaign, and probably the beginning of a new round of inference cost competition.

Today marks the release of Gemini 3 Pro, the new state-of-the-art model.

Do yourself a favor: take a moment to explore the links below and try out the tools to see what this model can really do.

1. Gemini 3: Introducing the latest Gemini AI model from Google

2. Start building with Gemini 3

3. Gemini 3 brings upgraded smarts and new capabilities to the Gemini app

4. Google Antigravity

5. Generative UI: A rich, custom, visual interactive user experience for any prompt

Subscribe now

In today’s issue:

• AIGC Papers – π*0.6 brings real-world experience into home robots, “Back to Basics” rethinks diffusion as true denoising, and Depth Anything 3 offers a cleaner geometric backbone for future world-models.

• AIGC Projects – Depth Anything 3 lands in ComfyUI, AgentEvolver lets LLM agents train themselves, and a new Qwen upscaling LoRA targets real-world photography.

• AIGC News – Gemini 3 Pro’s powerful model card leaks, xAI rolls out Grok 4.1, and Replicate joins Cloudflare right as a major outage hits the platform.

Today’s AIGC Papers

• π*0.6 shows how a VLA can truly learn from experience, turning home robots into durable, all-day performers through demonstrations, corrections, and self-practice.

• Back to Basics pulls diffusion models back to real denoising, using a pure image-space Transformer to achieve high-fidelity generation with a simpler formulation.

• Depth Anything 3 reconstructs consistent 3D geometry from any view using a single Transformer, giving future world-models a more stable and cleaner geometric backbone.

1. π*0.6: a VLA that Learns from Experience( blog | paper )

   Physical Intelligence presents π*0.6, a VLA combined with advantage-conditioned RL, trained through demonstrations, corrections, and self-experience. The system doubles success rates and throughput on real household tasks—making coffee, folding clothes, packing objects—and supports long, continuous, real-world operation, and the model card for π0.6 is also provided in the paper.

2. Kaiming He just Introduced JiT! Back to Basics: Let Denoising Generative Models Denoise（ paper | code)

   MIT’s Li Tianhong and Kaiming He propose predicting the clean image instead of noise, using a large patch-based ViT (“Just-image Transformers”) in pixel space. The method needs no tokenizer or pretraining yet achieves competitive high-res ImageNet generation, offering a simpler and clearer theoretical view of diffusion.

3. Depth Anything 3: Recovering the Visual Space from Any Views ( webpage | paper | code )

   ByteDance introduces Depth Anything 3, using a single Transformer and depth-ray representation to recover consistent geometry from single images, multi-view inputs, or videos. It surpasses VGGT in camera-pose and geometry accuracy, and outperforms DA2 in monocular depth, enabling high-fidelity 3DGS reconstruction.

Today’s AIGC Projects

• ComfyUI-DepthAnythingV3 turns Depth Anything 3 into drop-in ComfyUI nodes, making DA3 usable in everyday image and video workflows.

• AgentEvolver provides a self-evolving training loop where LLM agents generate their own tasks, rewards, and improvements.

• Qwen-Edit-2509-Upscale-LoRA gives Qwen-Image-Edit a practical, detail-preserving photography upscaler for real-world enhancement.

1. ComfyUI-DepthAnythingV3 ( link )

   ComfyUI-DepthAnythingV3 wraps Depth Anything 3 into ready-made ComfyUI nodes, letting users run DA3 on images and, with custom graphs, on multi-view / video inputs. It exposes DA3’s spatially consistent depth prediction in a visual workflow, so you can feed depth maps directly into ControlNet, 3DGS, or other geometry-aware pipelines.

2. AgentEvolver: Towards Efficient Self-Evolving Agent System ( link )

   AgentEvolver is an end-to-end framework where LLM-based agents teach themselves via three loops: self-questioning (generate new tasks), self-navigating (reuse past trajectories with hybrid policies), and self-attributing (fine-grained credit assignment over states/actions. It cuts dataset engineering cost, improves exploration efficiency, and yields faster capability gains than traditional RL-style agent training.

3. vafipas663/Qwen-Edit-2509-Upscale-LoRA ( link )

   Qwen-Edit-2509-Upscale-LoRA is a LoRA adapter for Qwen-Image-Edit-2509 focused on realistic photography. Trained on UltraHR-100K and Unsplash-lite, it repairs extreme low resolution, oversharpening, strong JPEG artifacts, motion blur, pixelation, and heavy noise—often up to 16×—while preserving composition and structure, making it a practical replacement for many “magic” commercial upscalers.

Today’s AIGC News

• Gemini 3 Pro model card leak – A leaked model card outlines Google’s next-gen sparse MoE multimodal model with 1M context, 64K outputs, RL training, detailed agent benchmarks, safety evals, and a Jan 2025 knowledge cutoff.

• Grok 4.1 announced – xAI unveils Grok 4.1, a new iteration of the Grok family with upgraded capabilities and overall performance.

• Replicate × Cloudflare, amid outage – Replicate is joining Cloudflare just as Cloudflare suffers a major service outage impacting large parts of its platform.

1. The model card for Gemini 3 Pro has reportedly leaked, revealing a model with extremely strong capabilities

   Leaked Gemini 3 Pro model card describing Google’s next-generation sparse MoE multimodal model with 1M-token context, 64K outputs, trained on large-scale web, code and media with RL. It details agentic performance, deployment channels, benchmarks, safety evaluations, frontier safety status, remaining risks, and a January 2025 knowledge cutoff.

2. xAI has announced Grok 4.1, a new iteration of its Grok model family with upgraded capabilities and performance. ( link )

3. Replicate is joining Cloudflare, at the same time that Cloudflare has been experiencing a significant service outage affecting its platform. ( link )

Subscribe now

Always fresh, always live

Real-time AIGC tracker

New models, papers, and projects as they drop — stay ahead of the AI curve.

For deeper insights and long-form analysis, subscribe to our weekly briefings at newsletter.aigc.news.

🚀 See today’s AIGC highlights

That’s it for today.

Keep building, keep thinking for yourself — we’ll be here tracking the next wave.

The aigc.news Team

My last update here was several months ago. During this break, I’ve been focused on one simple question:

In this era of exploding AIGC information, what can I offer that is genuinely useful?

I’ve realized that people don’t need more information. What’s missing are two key things:

1. Someone to filter the noise for you.

2. A way to quickly know “the few things worth seeing today.”

So starting today, I’m tackling this with a much simpler approach.

1. aigc.news: 3 News + 3 Papers + 3 Projects, Every Day

Over the past few months, I’ve been quietly building aigc.news.

It will now be updated daily and is completely free. Each day, you’ll find just three essential sections:

• 3 AIGC News: The developments that actually matter, with one-click access to the original sources.

• 3 AIGC Papers: A single sentence explaining why each one is worth your attention.

• 3 AIGC Projects : Who they’re for and what they do, at a glance.

I’m also building two small extensions:

• A “3x3 daily signal card” that’s easy to save and share.

• A subdomain: live.aigc.news, acting as a simple live radar for paper and project updates.

Think of aigc.news as your: “1-minute scan of the 9 must-see signals in the AIGC world today.”

2. newsletter.aigc.news: Weekly Digests + Occasional Deep Dives

This newsletter will now focus on two things:

• Weekly Digest From the week’s daily signals, I’ll pull out the ~10 most worth revisiting and add my perspective: what’s just hype, and what might shape the coming months.

• Occasional In-Depth Essays When a key question surfaces repeatedly, I’ll explore it in a long-form article you can reference later. For example:

  • What is the true division of labor between open-source and closed-source?

  • How do multimodal models, embodied intelligence, and agents fit together?

  • What opportunities remain for indie developers and small teams in this wave?

In short: aigc.news gives you the daily signals. newsletter.aigc.news helps you see the broader direction.

3. Subscriptions & Pricing (For Now)

It’s very simple:

• aigc.news: Free daily updates.

• newsletter.aigc.news: The weekly digest and most long-form pieces will also be free.

Your subscription status won’t change, and you won’t lose access to content.

My priority right now is to get this system running smoothly and prove its usefulness to you. Everything else can wait.

The AIGC space is noisy enough. I have no intention of adding to it. I just want to do one thing well:'

With a steady, sustainable rhythm, help you notice the few signals that truly matter.

Welcome back. I hope that starting today, you’ll make aigc.news your quick morning radar.

— pxiaoer

Subscribe now

Top Papers of the week

1.) Tracing the thoughts of a large language model ( blog | paper1 | paper2 )

Anthropic's research on the Claude language model reveals key insights:

• Multilingual Ability: Claude uses a shared conceptual space across languages, enabling knowledge transfer and suggesting a universal "language of thought."

• Poetry: It plans ahead for rhyming, showing foresight and flexibility.

• Mental Math: Claude combines approximation and precise calculation to solve problems, reflecting complex internal strategies.

• Reasoning: It performs multi-step reasoning by integrating facts, demonstrating adaptability.

• Hallucinations: Claude avoids guessing to reduce hallucinations but can still falter in some cases.

• Jailbreaks: Specific prompts can bypass safety mechanisms, exploiting coherence-safety conflicts.

2.) Synthetic Video Enhances Physical Fidelity in Video Synthesis ( webpage | paper )

We explore enhancing video generation models using physics-consistent synthetic videos from computer graphics. These videos maintain 3D consistency and improve model fidelity by reducing artifacts. Our method curates synthetic data and transfers its realism, boosting physical consistency across tasks. While not fully understanding physics, this work shows synthetic videos can enhance physical fidelity in video synthesis.

3.) Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback ( paper )

RLHF is essential for aligning large language models with human preferences, but prompt-data construction has been neglected. This paper explores data-driven bottlenecks in RLHF performance scaling, focusing on reward hacking and reduced response diversity. We propose a hybrid reward system combining RTV and GenRM to counter reward hacking and introduce Pre-PPO to maintain response diversity and boost learning efficiency. Prioritizing math and coding tasks early in training also significantly improves performance. Experiments on two model sizes show that RTV is most resistant to reward hacking, followed by GenRM with ground truth and then GenRM with SFT Best-of-N responses. Our methods capture task-specific nuances quickly, enhancing overall RLHF performance. This work highlights the importance of careful data construction and provides practical solutions to overcome performance barriers in RLHF.

4.) Gemini Robotics: Bringing AI into the Physical World ( paper )

Recent advancements in multimodal models have shown promise in digital domains, but translating these capabilities to physical robots remains challenging. This report introduces Gemini Robotics, a new family of AI models built on Gemini 2.0, specifically designed for robotics. Gemini Robotics is a Vision-Language-Action (VLA) model that can directly control robots, performing complex manipulation tasks with smooth, reactive movements. It is robust to variations in objects and environments and can follow diverse instructions. With fine-tuning, it can tackle long-horizon tasks, learn new tasks from few demonstrations, and adapt to novel robot embodiments. This is enabled by Gemini Robotics-ER, an extended model that enhances spatial and temporal reasoning for robotics tasks such as object detection, trajectory prediction, and 3D bounding box predictions. The Gemini Robotics family represents a significant step towards general-purpose robots, addressing safety considerations and unlocking AI's potential in the physical world.

5.) Qwen2.5-Omni Technical Report( paper )

We present Qwen2.5-Omni, an end-to-end multimodal model handling text, images, audio, and video inputs while generating text and speech responses in a streaming manner. It uses block-wise processing for audio and visual inputs, synchronized via TMRoPE. The Thinker-Talker architecture separates text (Thinker) and speech (Talker) generation to prevent interference, with Talker using sliding-window DiT for low-latency audio decoding. Qwen2.5-Omni outperforms Qwen2-Audio, matches Qwen2.5-VL, and sets new benchmarks on Omni-Bench, excelling in speech instruction following, robustness, and naturalness.

6.) Scaling Laws of Synthetic Data for Language Models ( paper )

Large language models (LLMs) rely on high-quality web data, but this resource is depleting. Synthetic data offers a solution, though its scalability remains uncertain. We propose SynthLLM, a framework that creates high-quality synthetic datasets by recombining concepts from pre-training corpora. Key findings include: (1) SynthLLM follows scaling laws reliably, (2) performance plateaus at 300B tokens, and (3) larger models need fewer tokens to optimize. SynthLLM outperforms existing methods, proving synthetic data as a scalable alternative for advancing LLM performance.

7.) GAIA-2: Pushing the Boundaries of Video Generative Models for Safer Assisted and Automated Driving( blog | paper )

Generative models enable environment simulation but lack key features for autonomous driving, like multi-agent interactions and multi-camera consistency. We present GAIA-2, a latent diffusion model that generates controllable, high-resolution, spatiotemporally consistent videos across diverse driving environments. GAIA-2 integrates structured inputs and latent embeddings to simulate complex, scalable driving scenarios, advancing autonomous system development.

8.) ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Model( webpage | paper )

Real-time interactive video-chat portraits are advancing but struggle with synchronized body motions and fine control over expressions. We propose a framework for stylized video generation, extending from talking heads to upper-body interaction. Using hierarchical motion diffusion and explicit hand control, our method generates expressive, synchronized videos at 512×768 resolution, 30fps, enabling real-time, natural video chats with rich gestures and realism.

9.) What, How, Where, and How Well? A Survey on Test-Time Scaling in Large Language Models ( paper )

As pretraining-era scaling wanes, test-time scaling (TTS) has gained focus, enhancing LLMs' problem-solving in tasks like math, coding, and open-ended Q&A. This survey introduces a unified framework across four TTS dimensions: what, how, where, and how well to scale. We review methods, applications, and challenges, offering deployment guidelines and future directions for further scaling and broader generalization.

10.) Large Language Model Agent: A Survey on Methodology, Applications and Challenges( paper | repo )

The rise of intelligent agents, powered by large language models (LLMs), marks a step toward artificial general intelligence. This survey presents a taxonomy of LLM agents, exploring their architecture, collaboration, evolution, and emergent behaviors in complex environments. It unifies fragmented research, examines evaluation methods, tools, challenges, and applications, and highlights future research directions.

AIGC News of the week

1.) deepseek-ai/DeepSeek-V3-0324 ( huggingface )

2.) bytedance’s MegaTTS 3 ( repo )

3.) OpenAI Agents SDK support MCP ( link )

4.) Gemini 2.5: Google’s most intelligent AI model ( link )

5.) VGGT: Visual Geometry Grounded Transformer ( repo )

more AI News: live.aigc.news

Inference System Design Principles

DeepSeek first introduced the design principles of the inference system, with optimization goals focused on: greater throughput and lower latency.

The solution currently adopted by DeepSeek: Large-Scale Node Expert Parallelism (EP)

• EP significantly increases batch size, thereby improving the efficiency of GPU matrix multiplication and boosting throughput.

• EP distributes experts across different GPUs, with each GPU only needing to compute a small number of experts (thus reducing memory access demands), lowering latency.

Introduced Complexity:

• EP introduces cross-node transmission. To optimize throughput, a suitable computation workflow must be designed to allow transmission and computation to occur synchronously.

• EP involves multiple nodes, naturally requiring Data Parallelism (DP), with load balancing needed between different DP instances.

Large-Scale Cross-Node Expert Parallelism

DeepSeek employs a multi-machine, multi-card expert parallelism strategy:

• Prefill: Router Expert EP32, MLA, and Shared Expert DP32. One deployment unit consists of 4 nodes, 32 redundant router experts, with 9 router experts and 1 shared expert per card.

• Decode: Router Expert EP144, MLA, and Shared Expert DP144. One deployment unit consists of 18 nodes, 32 redundant router experts, with 2 router experts and 1 shared expert per card.

The multi-machine, multi-card expert parallelism introduces significant communication overhead, so dual-batch overlapping is used to mask communication costs and improve overall throughput.

• In the prefill phase, computation and communication of two batches are interleaved—one batch’s computation can mask the communication overhead of the other.

• In the decode phase, execution times vary across stages, so the attention part is split into two stages, creating a 5-stage pipeline to overlap computation and communication.

Due to the use of large-scale parallelism (including expert parallelism and data parallelism), some GPUs may become overloaded, necessitating computational and communication load balancing.

• Prefill Load Balancer

  • Core Issue: Variations in the number and length of requests across different Data Parallelism (DP) instances lead to differences in core-attention computation and dispatch transmission volumes.

  • Optimization Goal: Ensure roughly equal computation load across GPUs (core-attention load balancing) and similar token input volumes (dispatch transmission load balancing) to avoid prolonged processing times on some GPUs.

• Decode Load Balancer

  • Core Issue: Variations in the number and length of requests across DP instances result in differences in core-attention computation (related to KVCache usage) and dispatch transmission volumes.

  • Optimization Goal: Ensure roughly equal KVCache usage across GPUs (core-attention load balancing) and similar request volumes (dispatch transmission load balancing).

• Expert-Parallel Load Balancer

  • Core Issue: For a given MoE model, certain naturally high-load experts exist, leading to uneven expert computation loads across GPUs.

  • Optimization Goal: Balance expert computation across GPUs (i.e., minimize the maximum dispatch reception volume across all GPUs).

Real-World Statistics of the Online Inference System

All DeepSeek R1/V3 services use H800 GPUs. Matrix computations and dispatch transmissions use FP8 format consistent with training, while core-attention computations and combine transmissions use BF16 consistent with training, maximizing service performance.

Since service loads are high during the day and low at night, all servers handle inference during peak load times, while some machines are freed up for research and training during low-load periods.

Over the past 24 hours (Beijing time, 2025/02/27 12:00 to 2025/02/28 12:00), the total node usage for DeepSeek V3 and R1 inference services peaked at 278 nodes and averaged 226.75 nodes (each node with 8 H800 GPUs). Assuming a GPU rental cost of $2/hour, the total daily cost is $87,072.

Within this 24-hour statistical period, DeepSeek V3 and R1 recorded:

• Total input tokens: 608 billion, of which 342 billion tokens (56.3%) hit the KVCache disk cache.

• Total output tokens: 168 billion. The average output rate was 20–22 tokens per second (tps), with an average KVCache length of 4,989 per output token.

• Average throughput per H800 GPU:

  • For prefill tasks: ~73.7k tokens/s input throughput (including cache hits).

  • For decode tasks: ~14.8k tokens/s output throughput.

These statistics encompass all loads from web, app, and API usage. If all tokens were priced according to DeepSeek R1’s rates, the theoretical daily revenue would be $562,027, yielding a cost-profit margin of 545%.

In reality, DeepSeek doesn’t generate this much revenue because V3’s pricing is lower, paid services only account for a portion of usage, and discounts are offered at night.

What Can We Infer?

Rumors previously suggested that DeepSeek’s official deployment consisted of a 320-H800 inference cluster. Now, it appears to be 278 nodes—2,224 H800s. Officially, DeepSeek acknowledges owning at least 10,000 H800s, meaning the GPUs used for inference are relatively few.

• Cost: Average of 226.75 nodes (1,814 GPUs), at $2/hour per GPU, yields a daily cost of $87,072.

• Revenue: Input: 608B tokens; Output: 168B tokens, resulting in a daily revenue of $562,027.

• Gross Daily Profit: $474,955 = ~3,457,672.4 RMB/day.

However, the above calculation has flaws. Even if the 6x profit margin is halved to 3x, the profit margin remains very high. Many domestic vendors deploying DeepSeek have shut down API services due to losses, raising questions about where the problem lies.

In an interview, Liang Wenfeng said: “We just do things at our own pace, then calculate costs and set prices. Our principle is not to lose money.”

This suggests DeepSeek is currently profitable, with earnings likely reinvested into R&D. We look forward to R2’s release soon.

Subscribe now

According to the official introduction, the Fire-Flyer File System (3FS) is a parallel file system designed to fully utilize the bandwidth of modern SSDs and RDMA networks.

• Achieves an aggregate read throughput of 6.6 TiB/s in a 180-node cluster.

• Delivers 3.66 TiB/minute throughput in the GraySort benchmark on a 25-node cluster.

• Provides peak throughput of over 40 GiB/s per client node in KVCache lookups.

• Features a distributed architecture with strong consistency semantics.

• Used for training data preprocessing, dataset loading, checkpoint saving/reloading, embedding vector search, and KVCache lookups in inference for V3/R1.

Smallpond, on the other hand, is a data processing framework built on top of 3FS.

Fire-Flyer File System (3FS)

3FS is part of the Fire-Flyer AI-HPC developed by DeepSeek. It is detailed in the paper Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning.

Fire-Flyer AI-HPC consists of three components: the HAI Platform (open-sourced two years ago), 3FS (open-sourced today), and HaiScale (yet to be open-sourced).

In summary, 3FS has several key features:

1. High-Performance Design: 3FS is tailored to leverage the high IOPS (input/output operations per second) and throughput of NVMe SSDs, as well as RDMA networks. This design enables it to efficiently handle large-scale data requests, meeting the demands of deep learning and large-scale computing.

2. System Architecture: The 3FS system comprises four roles: cluster manager, metadata service, storage service, and client. The metadata and storage services periodically send heartbeat signals to the cluster manager to ensure system stability and efficiency. Multiple cluster managers ensure high availability.

3. Request Control Mechanism: 3FS implements a request transmission control mechanism to alleviate network congestion. Upon receiving a read request, the storage service asks the client for permission to transfer data. This limits the number of concurrent senders, maintaining good performance under high load.

4. Strong Consistency with Chain Replication: 3FS adopts the Chain Replication and Allocate Query (CRAQ) approach to provide strong consistency. File contents are split into blocks and replicated across a series of storage targets, fully unleashing the throughput and IOPS of all SSDs.

5. High Throughput: By optimizing batch write and read operations, 3FS achieves write speeds exceeding 10 GiB/s per node, accelerating checkpoint saving and loading, and reducing latency during training.

6. 3FS-KV System: 3FS also supports 3FS-KV, a shared-storage distributed data processing system built on 3FS. It supports key-value storage, message queues, and object storage models, further enhancing system flexibility and performance.

3FS provides robust storage support for deep learning and large-scale computing, effectively meeting demands for high throughput and low latency.

Description from the Paper:

Performance

Peak Throughput

Read throughput test results for a 3FS cluster: The cluster consists of 180 storage nodes, each equipped with 2×200Gbps InfiniBand NICs and 16×14TiB NVMe SSDs. Over 500+ client nodes, each with a 1×200Gbps InfiniBand NIC, were used for the read stress test. Under background traffic from training jobs, the aggregate read throughput reached approximately 6.6 TiB/s.

Sorting Performance

The test cluster consists of 25 storage nodes (2 NUMA domains per node, 1 storage service per NUMA, 2×400Gbps NICs per node) and 50 compute nodes (2 NUMA domains, 192 physical cores, 2.2 TiB RAM, and 1×200Gbps NIC per node). Sorting 110.5 TiB of data across 8192 partitions took 30 minutes and 14 seconds, achieving an average throughput of 3.66 TiB/minute.

KVCache

KVCache is designed to optimize the LLM inference process by caching keys and value vectors from previous tokens in the decoder layers, avoiding redundant computation. The figure above shows the read throughput for all KVCache clients, highlighting peak and average values, with a peak throughput of up to 40 GiB/s. The figure below shows the IOPS of delete operations during garbage collection (GC) over the same period.

Why is a Specialized File System Like 3FS Needed?

In LLM scenarios, there’s a need for highly concurrent, high-throughput, and scalable distributed file systems that also demand strong consistency, intelligent routing, and cache management. Systems like 3FS, which offer high-performance solutions tailored for RDMA and SSDs, flexible metadata design, and asynchronous zero-copy I/O, become highly valuable.

Code Analysis

One notable aspect of 3FS’s implementation is that DeepSeek used Rust to develop the chunk_engine.

The chunk_engine is a core module at the bottom layer of the 3FS storage service, responsible for managing, allocating, and reclaiming physical disk blocks. The upper layers can read and write block data through this engine. It primarily uses cxx to automatically generate C++ bindings, allowing C++ code to directly call Rust code.

In recent years, Rust has gained popularity in the MLSys (Machine Learning Systems) field. For example, Hugging Face’s tokenizers are also implemented in Rust. The DeepSeek team likely chose Rust for the chunk_engine due to its maintainability, memory safety, and excellent performance.

The DeepSeek team may also have used the Rust framework Tokio in backend services, as I found several Rust open-source projects, including Tokio, in Quant AI’s open-source initiatives. I sincerely hope more teams adopt Rust for developing machine learning systems.

Smallpond

Smallpond is a lightweight data processing framework built on top of DuckDB and 3FS. It supports lightweight, high-performance data processing and scales to petabyte-scale datasets.

Installation and usage are straightforward, with a minimal API offering two types: one for dynamically building dataflow graphs and another for static construction.

Installation:

pip install smallpond

Usage Example:

# Download example data
wget https://duckdb.org/data/prices.parquet

import smallpond
# Initialize session
sp = smallpond.init()
# Load data
df = sp.read_parquet("prices.parquet")
# Process data
df = df.repartition(3, hash_by="ticker")
df = sp.partial_sql("SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker", df)
# Save results
df.write_parquet("output/")
# Show results
print(df.to_pandas())

For performance, refer to the sorting performance of 3FS.

Conclusion of DeepSeek Open Source Week

DeepSeek Open Source Week concludes today. Thank you, DeepSeek, for sharing valuable resources for everyone to learn and use.

Many teams have already taken action and achieved tangible performance improvements. For instance, the vLLM team replaced TRITON_MLA with FLASHMLA, boosting throughput by 2-16%, delivering real results.

These projects open-sourced by DeepSeek will continue to influence us. Our journey of intense learning continues.

more

• day0: https://aigc.openbot.ai/p/deepseek-opensourceweek-is-coming

• day1: https://aigc.openbot.ai/p/deepseek-open-source-week-day-1-in

• day2:https://aigc.openbot.ai/p/day-2-of-deepseek-opensourceweek

• day3:https://aigc.openbot.ai/p/deepseek-opensourceweek-day-3-deepgemm

• day4:https://aigc.openbot.ai/p/deepseek-opensourceweek-day-4-in

Subscribe now

DualPipe is a bidirectional pipeline parallelism algorithm designed for computation-communication overlap in V3/R1 training. Meanwhile, EPLB serves as an expert-parallel load balancer for V3/R1.

The final project, profile-data, primarily releases analytical data from DeepSeek’s infrastructure for training and inference. So far, it includes data on Prefilling for both training and inference, while the Decoding analysis data for inference has yet to be made public.

Today, we’ll dive into an analysis of the DualPipe and EPLB projects, both of which lean toward engineering optimization.

DualPipe

DualPipe is mentioned in the DeepSeek V3 paper as a bidirectional pipeline parallelism communication algorithm, mainly used to optimize data interaction and training efficiency in large-scale models.

Key Features:

• Computation-Communication Overlap
  DualPipe’s design aims to maximize cluster computing performance by achieving full overlap of computation and communication during forward and backward passes, reducing idle wait times typical in traditional pipeline parallelism. This is especially critical for expert parallelism (Expert Parallelism) across nodes in MoE models.

• Bidirectional Scheduling
  DualPipe employs a bidirectional scheduling strategy, feeding data from both ends of the pipeline simultaneously to reuse hardware resources efficiently. It also incorporates a sophisticated yet highly effective 8-step scheduling strategy.

• Memory Optimization
  DualPipe deploys the shallowest layers (including the embedding layer) and the deepest layers (including the output layer) on the same pipeline level (PP Rank), enabling physical sharing of parameters and gradients to further enhance memory efficiency.

Pipeline Bubble and Memory Usage Comparison (Pipeline bubble refers to idle wait time)

For those interested, you can check out the code—it’s not very long and is great for learning.

EPLB

EPLB (Expert Parallelism Load Balancer) is primarily designed to optimize the distributed deployment of MoE models. It ensures load balancing among different experts in the MoE portion by replicating shared experts and fine-grained high-load experts across multiple GPUs in the cluster. This allows GPUs to handle more "hot data" (data sent to shared experts) efficiently.

EPLB isn’t detailed in DeepSeek’s paper, and its code is remarkably concise at just 160 lines.

Key Features:

• Load Balancing Optimization
  It replicates high-load experts (a strategy we can call "redundant expert strategy") and uses heuristic adjustments for expert allocation to ensure balanced workloads across GPUs.

• Hierarchical Load Balancing
  EPLB adopts a three-tier structure: node-level → intra-node expert replication → GPU allocation. It prioritizes assigning experts from the same group to the same node to minimize cross-node data transfers, then ensures load balancing at each layer. This approach, combined with DeepSeek V3’s Group-Limited Expert Routing strategy, significantly boosts distributed training efficiency.

• Dynamic Scheduling Strategy
  EPLB dynamically selects load balancing strategies based on the situation—using a hierarchical strategy during the prefilling phase and a global strategy during the decoding phase.

Let’s Look at the Code:

Redundant Expert Strategy

def replicate_experts(weight: torch.Tensor, num_phy: int):
    # Replicate high-load experts
    for i in range(num_log, num_phy):
        redundant_indices = (weight / logcnt).max(dim=-1).indices
        phy2log[:, i] = redundant_indices
        logcnt[arangen, redundant_indices] += 1

Hierarchical Load Balancing

def rebalance_experts_hierarchical():
    # Step 1: Pack expert groups to nodes
    tokens_per_group = weight.unflatten(-1, (num_groups, group_size)).sum(-1)
    group_pack_index, group_rank_in_pack = balanced_packing(tokens_per_group, num_nodes)

    # Step 2: Build redundant experts within nodes
    tokens_per_mlog = weight.gather(-1, mlog2log).view(-1, num_logical_experts // num_nodes)

    # Step 3: Pack physical experts to GPUs
    tokens_per_phy = (tokens_per_mlog / mlogcnt).gather(-1, phy2mlog)

Dynamic Scheduling Strategy

def rebalance_experts():
    if num_groups % num_nodes == 0:
        # Use hierarchical strategy
        phy2log, phyrank, logcnt = rebalance_experts_hierarchical()
    else:
        # Use global strategy
        phy2log, phyrank, logcnt = replicate_experts()

Interested readers can explore the full code.

Tomorrow is the final day of DeepSeek Open Source Week—will they drop a heavyweight open-source project? Let’s wait and see!

Subscribe now

Today marks the third day of DeepSeek's Open Source Week, with the release of DeepGEMM right on schedule at 9 AM.

As of now, the project has garnered 3.3k stars since its release.

The official introduction describes DeepGEMM as an FP8-supporting GEMM library compatible with both dense and MoE (Mixture of Experts) GEMM operations, designed for training and inference of V3/R1 models.

A Brief Introduction to GEMM

General Matrix Multiplication (GEMM) is one of the most fundamental and critical operations in deep learning and scientific computing. GEMM refers to the multiplication of two matrices, A and B, to produce a result matrix C, typically expressed as C = A × B.

In deep learning, GEMM underpins core components such as fully connected layers, convolutional layers, and attention mechanisms. For instance, in Transformer architectures, both self-attention and feedforward network layers heavily rely on matrix multiplication. As model sizes grow, GEMM operations dominate the computational time in training and inference, making their performance a key factor in the efficiency of deep learning systems.

Modern GPU architectures, like NVIDIA’s Tensor Core technology, are specifically designed to accelerate matrix multiplication. With the ever-increasing scale of models, the demand for high-performance GEMM implementations continues to rise, especially in large language models (LLMs) and MoE frameworks, where efficient GEMM is critical for real-time inference and cost-effective training.

In the paper DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, DeepSeek mentions GEMM, though it ties into their work in another paper, Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning, which introduces the HAI-LLM training system. For those interested, I recommend checking out the Fire-Flyer AI-HPC paper.

Today’s open-sourced DeepGEMM supports FP8 and is tailored for training and inference of DeepSeek’s V3/R1 models. In the V3 paper, DeepSeek details several optimizations for FP8 training.

The main challenges of FP8 training lie in precision and error handling. To tackle these, DeepSeek implemented the following optimizations:

• Fine-Grained Quantization: Data is split into smaller groups, each with a specific multiplier to maintain high precision.

• Online Quantization: Weights are computed online for each 1x128 activation block or 128x128 weight block, with scaling factors inferred on-the-fly, and activations converted to FP8 in real time.

• Improved Accumulation Precision: FP8 accumulation can introduce random errors, so intermediate results are stored in FP32, then converted back after accumulation.

• Low-Precision/Mixed-Precision Storage and Communication: For MoE model training, FP8 is mixed with BF16/FP32 to ensure dynamic model stability.

For a detailed look at these optimizations, check out the DeepSeek V3 paper.

DeepGEMM Overview

Here’s a summary of its key features:

• FP8 Support: DeepGEMM uses CUDA’s two-stage accumulation to address precision issues.

• Grouped GEMM Support: It improves on CUTLASS’s grouped GEMM, with targeted optimizations for MoE models.

• Just-In-Time Compilation: Through JIT technology, code is dynamically generated and optimized at runtime, boosting performance and flexibility.

• FFMA SASS Interleaving: DeepSeek analyzed SASS compilation results in depth, tweaking FFMA/FADD instructions to enhance fine-grained FP8 GEMM efficiency.

Performance

All metrics show improvement, with the highest gain reaching a 2.7x speedup. DeepSeek notes that performance isn’t optimal in some areas and welcomes PRs from those interested in further optimization.

In the DeepGEMM project’s README, the DeepSeek team provides a detailed breakdown of the optimizations. For those interested, it’s worth diving into the code alongside the documentation for a hands-on exploration.

Spotlight: The interleave_ffma.py File

Today, let’s focus on a specific file in the project: interleave_ffma.py under the jit directory. It contains some clever tricks worth exploring.

Here’s the code:

import argparse
import mmap
import os
import re
import subprocess
from torch.utils.cpp_extension import CUDA_HOME

def run_cuobjdump(file_path):
    command = [f'{CUDA_HOME}/bin/cuobjdump', '-sass', file_path]
    result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    assert result.returncode == 0
    return result.stdout

def extract_ffma(sass):
    lines = sass.splitlines()
    collected = []
    current = []
    arch_name, func_name = 'N/A', 'N/A'
    skip_next_line = False
    for line in lines:
        if 'code for' in line:
            arch_name = line.lstrip().lstrip('code for ').rstrip()
        elif 'Function :' in line:
            func_name = line.lstrip().lstrip('Function :').rstrip()
        elif 'FFMA' in line:
            current.append(line)
            skip_next_line = True
        elif skip_next_line:
            current.append(line)
            skip_next_line = False
        else:
            if len(current) >= 16:
                assert len(current) % 2 == 0
                collected.append((f'{arch_name}::{func_name}', current))
            current = []
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f"Found {len(collected)} FFMA segments")
    return collected

def extract_hex_from_line(line):
    match = re.search(r'/\*\s*(0x[0-9a-fA-F]+)\s*\*/', line)
    assert match
    return int(match.group(1), 16)

def validate(m, offset, le_bytes, num_lines):
    assert len(le_bytes) == num_lines // 2
    assert m[offset:offset + 16] == le_bytes[0]
    for i in range(1, num_lines // 2):
        if m[offset + i * 16:offset + i * 16 + 16] != le_bytes[i]:
            return False
    return True

def parse_registers(line):
    import re
    line = re.sub(r'/\*.*?\*/', '', line)
    line = line.replace(';', '')
    tokens = line.strip().split(',')
    registers = []
    for token in tokens:
        token = token.strip()
        words = token.split()
        for word in words:
            if word.startswith('R'):
                reg = word.split('.')[0]
                registers.append(reg)
    return registers

def modify_segment(m, name, ffma_lines):
    num_lines = len(ffma_lines)
    assert num_lines % 2 == 0
    le_bytes, new_le_bytes = [], []
    reused_list = []
    dst_reg_set = set()
    last_reused, last_dst_reg = False, ''
    num_changed = 0
    for i in range(num_lines // 2):
        dst_reg = parse_registers(ffma_lines[i * 2])[-2]
        low_line, high_line = ffma_lines[i * 2], ffma_lines[i * 2 + 1]
        low_hex, high_hex = extract_hex_from_line(low_line), extract_hex_from_line(high_line)
        le_bytes.append(low_hex.to_bytes(8, 'little') + high_hex.to_bytes(8, 'little'))
        reused = (high_hex & 0x0800000000000000) != 0
        if reused:
            is_first_occurred = dst_reg not in dst_reg_set
            if is_first_occurred or (last_reused and dst_reg == last_dst_reg):
                assert high_hex & 0x0800200000000000, f"{hex(high_hex)}"
                high_hex ^= 0x0800200000000000
                reused = False
                num_changed += 1
            else:
                reused_list.append(i)
        dst_reg_set.add(dst_reg)
        new_le_bytes.append(low_hex.to_bytes(8, 'little') + high_hex.to_bytes(8, 'little'))
        last_reused, last_dst_reg = reused, dst_reg
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f" > segment `{name}` new reused list ({num_changed} changed): {reused_list}")
    offsets = []
    offset = m.find(le_bytes[0])
    while offset != -1:
        offsets.append(offset)
        offset = m.find(le_bytes[0], offset + 1)
    offsets = list(filter(lambda x: validate(m, x, le_bytes, num_lines), offsets))
    for offset in offsets:
        for i in range(num_lines // 2):
            m[offset + i * 16:offset + i * 16 + 16] = new_le_bytes[i]

def process(path):
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f'Processing {path}')
    output = run_cuobjdump(path)
    segments = extract_ffma(output)
    with open(path, 'r+b') as f:
        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE)
        for segment in segments:
            modify_segment(mm, *segment)
        mm.close()

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Interleave FFMA reg reuse')
    parser.add_argument('--so', help='Path to the SO file')
    args = parser.parse_args()
    process(args.so)

This file is designed to optimize the register reuse patterns of FFMA (Fused Floating-point Multiply-Add) instructions in CUDA-compiled assembly code by modifying the binary file, ultimately improving GPU instruction execution efficiency.

Key Function Breakdown:

SASS Code Extraction

def run_cuobjdump(file_path):
    command = [f'{CUDA_HOME}/bin/cuobjdump', '-sass', file_path]

Uses NVIDIA’s cuobjdump tool to extract SASS (assembly) code from the binary.

FFMA Instruction Analysis

def extract_ffma(sass):

Extracts segments of SASS code containing FFMA instructions, collecting architecture and function names along with the instruction sequences.

Register Usage Analysis

def parse_registers(line):

Parses the registers used in each instruction, identifying those starting with 'R'.

Binary Modification

def modify_segment(m, name, ffma_lines):

Modifies the reuse and yield bits of FFMA instructions by tweaking specific bit patterns (e.g., 0x0800200000000000) to optimize register reuse.

Workflow:

• Reads a compiled CUDA shared library (.so file).

• Extracts SASS code using cuobjdump.

• Identifies and collects all FFMA instruction sequences.

• Analyzes register usage patterns in each FFMA instruction.

• Modifies the reuse flags based on specific rules.

• Writes the optimized instructions back to the original file.

Optimization Strategy:

The tool targets:

• First-time register usage.

• Consecutive reuse of the same destination register.
  By tweaking the reuse and yield bits, it optimizes instruction scheduling.

Usage:

python interleave_ffma.py --so path/to/cuda_lib.so

Analysis:

This tool acts as a post-processing optimizer, running after CUDA compilation to enhance GPU instruction efficiency by tweaking the binary. Its focus on FFMA instruction register reuse is particularly impactful for compute-intensive applications like deep learning.

A regex in the file often stumps readers:

def extract_hex_from_line(line):
    match = re.search(r'/\*\s*(0x[0-9a-fA-F]+)\s*\*/', line)
    assert match
    return int(match.group(1), 16)

In CUDA SASS assembly, instructions often appear like this:

FFMA R8, R8, R6, R4;                  /* 0x5c98078000870808 */

This regex extracts 0x5c98078000870808, the hexadecimal machine instruction encoding. The function:

• Extracts the hex code from the assembly line.

• Converts it to an integer for subsequent modification.

• This step is crucial for locating instructions, modifying specific bits (e.g., reuse and yield flags), and writing them back.

Honestly, DeepSeek’s engineers seem to outshine even some NVIDIA folks when it comes to CUDA mastery! Oh, and they’ve slashed their API prices again.

Subscribe now

On the second day of OpenSourceWeek, the official DeepSeek X account posted an article at 10:24, introducing the second open-source project of Open Source Week: DeepEP.

Since the code was released, it has already garnered 4.3K stars.

Many people are not very familiar with MoE models, so this article will first briefly introduce MoE and some of DeepSeek's work on MoE.

MoE Introduction

The Mixture-of-Experts (MoE) model is a simple extension of the Transformer architecture, rapidly becoming the preferred architecture for medium-to-large-scale language models (2 billion to 600 billion parameters).

Key Advantages:

• Faster pre-training speed compared to dense models

• Faster inference speed compared to models with the same number of parameters

Challenges:
It requires significant memory since all expert systems need to be loaded into memory. Additionally, as it is typically used for medium-to-large models, it often requires parallel processing across multiple GPUs, and communication must be highly efficient.

MoE models primarily consist of two key components:

1. Sparse MoE Layers
   These layers replace the feed-forward network (FFN) layers in traditional Transformer models. An MoE layer contains several "experts" (e.g., 8), each of which is an independent neural network.

2. Gating Network or Router
   This component determines which tokens are sent to which experts. Sometimes, a token may even be routed to multiple experts.

Summary:
A notable advantage of Mixture-of-Experts (MoE) models is their ability to perform effective pre-training with far fewer computational resources than dense models. This means that, under the same computational budget, you can significantly scale up the model or dataset size. Especially during pre-training, MoE models typically reach the same quality level faster than dense models.

DeepSeek MoE

On January 11, 2024, DeepSeek released the paper DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, making it one of the earliest companies to research MoE models.

In the DeepSeekMoE paper, the concepts of Router Expert and Shared Expert were introduced, and experiments were conducted on increasing fine-grained experts.

In the paper, DeepSeekMoE 16B has 2 shared experts and 64 router experts per layer, with each token activating 2 shared experts and 6 router experts. The 145B version, on the other hand, has 4 shared experts and 128 router experts, with each token activating 4 shared experts and 12 router experts.

The DeepSeekMoE paper also mentions today’s topic: Expert Parallelism.

In the DeepSeek V3 paper, two paragraphs provide more detailed data. DeepSeek R1 was trained based on V3, and although no further details are provided, it is largely consistent with V3.

DeepEP Introduction

DeepEP is a communication library designed for MoE models and Expert Parallelism (EP). It provides high-throughput and low-latency all-to-all GPU kernels, also known as MoE data dispatch and combine. Additionally, the library supports low-precision operations, including FP8.

To align with the group-limited gating algorithm proposed in the DeepSeek-V3 paper, DeepEP offers a set of kernels optimized for asymmetric domain bandwidth forwarding (e.g., from NVLink domain to RDMA domain). These kernels deliver high throughput, making them ideal for training and inference prefetching tasks. They also support control over the number of Streaming Multiprocessors (SM).

For latency-sensitive inference decoding tasks, DeepEP includes a set of low-latency kernels using pure RDMA to minimize latency. The library also introduces a hook-based method for overlapping communication and computation without occupying any SM resources.

Note: The implementation of this library may slightly differ from the DeepSeek-V3 paper.

Performance

NVLink and RDMA Forwarding Test for General Kernels

We tested general kernels on H800 (NVLink max bandwidth ~160 GB/s), with each device connected to a CX7 InfiniBand 400 Gb/s RDMA NIC (max bandwidth ~50 GB/s). The test followed the pre-training setup of DeepSeek-V3/R1: 4096 tokens per batch, hidden dimension of 7168, top-4 grouping, top-8 experts, FP8 dispatch, and BF16 combine.

Low-Latency Kernel Test with Pure RDMA

We tested low-latency kernels on H800, with each device connected to a CX7 InfiniBand 400 Gb/s RDMA NIC (max bandwidth ~50 GB/s). The test followed a typical DeepSeek-V3/R1 production environment setup: 128 tokens per batch, hidden dimension of 7168, top-8 experts, FP8 dispatch, and BF16 combine.

Code Analysis

The optimization strategy focuses heavily on low-latency communication. Here, we’ll highlight this aspect, while other features like dynamic scheduling, asynchronous communication, and stream management can be explored in the relevant documentation and code.

Double Buffering

auto buffer = layout.buffers[low_latency_buffer_idx];
auto next_buffer = layout.buffers[low_latency_buffer_idx ^= 1];

• Alternates between two buffers: one for the current operation, another for the next.

• Uses the bitwise operation ^= 1 to efficiently switch buffer indices.

TMA (Tensor Memory Access) Optimization

• Leverages the Hopper architecture’s TMA instructions to accelerate data transfer.

• Supports low-precision formats like FP8 to reduce communication bandwidth requirements.

IBGDA Direct Communication

// Initialize recv queues for low-latency mode AR
ibgda_initialize_recv_queue<<<num_ranks, 128>>>(rank);

• Uses NVSHMEM’s IBGDA technology for GPU-direct RDMA communication.

• Bypasses CPU involvement entirely to reduce latency.

Expert-Level QP Allocation

_buffer = Buffer(group, 0, num_rdma_bytes, low_latency_mode=True,
num_qps_per_rank=num_experts // group.size())

• Assigns independent Queue Pairs (QPs) to each local expert, eliminating resource contention.

DeepEP Use Cases

• Large-scale MoE model training (e.g., models with hundreds of billions of parameters)

• High-concurrency, low-latency real-time inference services

• Heterogeneous computing tasks such as multimodal applications and scientific computing

Notably, at the end of the repository, DeepSeek mentions using an undocumented NVIDIA instruction for optimization—a true hacker spirit worth learning from!

Subscribe now

This morning at 9:34, DeepSeek announced the first project of Open Source Week on X: FlashMLA. This article provides an in-depth analysis of FlashMLA.

The FlashMLA project has gained significant popularity, with its code already reaching 6.8k stars.

Brief Introduction to MLA

MLA (Multi-Head Latent Attention) is an optimization method for Multi-Head Attention (MHA) proposed by DeepSeek in their paper DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model.

In Transformer models, MHA is one of the most computationally intensive modules. To maintain high efficiency in large-scale scenarios, further optimization is necessary.

MLA can be considered a variant of MHA. In its implementation, it borrows some concepts from FlashAttention. The DeepSeek-V2 paper primarily compares it with MHA, GQA, and MQA, with optimization results shown in the figure below:

In some inference frameworks, MLA has also been implemented. As shown below, after integrating MLA into SGLang, throughput increased by 2-3 times.

Using FlashMLA

Environment Requirements:

• Hopper GPUs

• Minimum CUDA 12.3

• Minimum PyTorch 2.0

Installation:

git clone https://github.com/deepseek-ai/FlashMLA  
python setup.py install  

Performance:
The repository provides a Benchmark file that can be run directly. Official results show that on an H800 SXM5 with CUDA 12.6, it achieves speeds of up to 3000 GB/s under memory-bound configurations and 580 TFLOPS under compute-bound configurations.

Code Analysis

FlashMLA’s codebase is relatively small with minimal dependencies.
The primary optimization techniques are as follows:

1. Computation Chunking and Scheduling Optimization

template<int kHeadDim_, int kBlockM_, int kBlockN_, int kNWarps_>  
struct Flash_fwd_kernel_traits_mla {  
    // Fixed block size of 64x64  
    static constexpr int kBlockM = kBlockM_;    
    static constexpr int kBlockN = kBlockN_;    
    
    // Each block uses 8 warps in parallel  
    static constexpr int kNWarps = kNWarps_;    
    static constexpr int kNThreads = kNWarps * 32;  
    
    // Shared memory optimization  
    static constexpr int kBlockKSmem = kHeadDim % 64 == 0 ? 64 : 32;  
};

Key Points:

• Improves computational efficiency through chunking (block size of 64), paged KV caching, and multi-warp parallelism.

2. Memory Access Optimization

struct Flash_fwd_mla_params {  
    using index_t = int64_t;  
    int b, seqlen_q, d, d_v;  
    int h, h_h_k_ratio, ngroups;  
    bool is_causal;  
    float scale_softmax, scale_softmax_log2;  
    int *__restrict__ cu_seqlens_k;  
    void *__restrict__ q_ptr;  
    void *__restrict__ k_ptr;  
    void *__restrict__ v_ptr;  
    void *__restrict__ o_ptr;  
    void *__restrict__ softmax_lse_ptr;  
    index_t q_batch_stride;  
    index_t k_batch_stride;  
    index_t v_batch_stride;  
    index_t o_batch_stride;  
    index_t q_row_stride;  
    index_t k_row_stride;  
    index_t v_row_stride;  
    index_t o_row_stride;  
    index_t q_head_stride;  
    index_t k_head_stride;  
    index_t v_head_stride;  
    index_t o_head_stride;  
    int *__restrict__ block_table;  
    index_t block_table_batch_stride;  
    int page_block_size;  
    int *__restrict__ tile_scheduler_metadata_ptr;  
    int num_sm_parts;  
    int *__restrict__ num_splits_ptr;  
    void *__restrict__ softmax_lseaccum_ptr;  
    void *__restrict__ oaccum_ptr;  
};

Key Points:

• Uses paged KV caching (block_table, page_block_size).

• Optimized memory layout and access strides (stride).

• Scheduling with tile_scheduler_metadata.

3. Softmax Computation Optimization

for (int mi = 0; mi < size<0>(tensor); ++mi) {  
    MaxOp<float> max_op;  
    max(mi) = zero_init ? tensor(mi, 0) : max_op(max(mi), tensor(mi, 0));  
    #pragma unroll  
    for (int ni = 1; ni < size<1>(tensor); ni++) {  
        max(mi) = max_op(max(mi), tensor(mi, ni));  
    }  
    max(mi) = Allreduce<4>::run(max(mi), max_op);  
    const float max_scaled = max(mi) == -INFINITY ? 0.f : max(mi) * scale;  
    sum(mi) = 0;  
    #pragma unroll  
    for (int ni = 0; ni < size<1>(tensor); ++ni)  {  
        tensor(mi, ni) = exp2f(tensor(mi, ni) * scale - max_scaled);  
        sum(mi) += tensor(mi, ni);  
    }  
    SumOp<float> sum_op;  
    sum(mi) = Allreduce<4>::run(sum(mi), sum_op);  
}

Key Points:

• Uses log2/exp2 instead of log/exp.

• Optimizes with FFMA instructions.

• Warp-level reduction for summation optimization.

4. Double Buffering Optimization

struct SharedStorageMLA {  
    union {  
        struct {  
            // Double buffering for K matrix  
            cute::array_aligned<Element, cosize_v<SmemLayoutQ>> smem_q;  
            cute::array_aligned<Element, cosize_v<SmemLayoutK> * 2> smem_k;  // Double buffer  
            cute::array_aligned<Element, cosize_v<SmemLayoutP>> smem_p;  
            cute::array_aligned<ElementAccum, cosize_v<SmemLayoutRow>> smem_scale;  
        };  
    };  
};

Key Points:

• Hides memory latency with double buffering to improve hardware utilization.

Summary of FlashMLA

FlashMLA is essentially a customized version of FlashAttention. Its current applicable scenarios include:

• Environments requiring CUDA 11+ and SM90+ Hopper architecture.

• Inference or training of multi-head attention with BF16 (Q=576, V=512).

• Large-sequence scenarios requiring integration with split-K schemes to boost throughput.

As shown above, there are still many optimization methods from the official team. Looking forward to tomorrow’s project—could it be infra-related? Perhaps something as challenging as MTP?

Subscribe now

On February 21, DeepSeek announced on X the launch of a warm-up for Open Source Week. Starting next week, they will open-source five projects over five consecutive days. For each of these five projects, I will write detailed articles to introduce them on the day they are announced. Feel free to follow me for the latest analyses.

Today, I’ll make some predictions about which projects might be open-sourced. If I guess even one correctly, I’ll consider it a win.

First, there’s definitely going to be something related to infra.
According to the X post, they are a small team within DeepSeek, sharing small but genuine progress, specifically modules used in online services. They particularly emphasized “small,” which likely points to code related to model inference optimization.

The recent release of DeepSeek-R1 has generated significant buzz, but inference optimization still lacks robust support from major frameworks. It feels like they might directly release some official implementations, as this is currently in high demand.

These optimizations could include deployment and inference strategies mentioned in the DeepSeek V3 technical report, such as Prefilling and Decoding.

Second, the official repository index hints at a paper, which is likely related.

Today, DeepSeek created an "open-infra-index" repo on GitHub, which includes a paper: Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning.

This Fire-Flyer AI-HPC paper mainly discusses optimizations for a training cluster with 10,000 A100 GPUs, focusing on reducing system construction costs and energy consumption. Specifically, compared to DGX-A100, it achieves about 80% of the performance while cutting costs by 60% and energy consumption by 50%. I won’t go into the detailed innovations here—interested folks can read the paper themselves.

Why mention this paper? My guess is that they’ll open-source some of the system code referenced in it. Looking at it, here are a few possibilities:

• HFReduce: A library developed specifically for efficient allreduce operations, designed to optimize GPU communication in PCIe architectures. HFReduce overlaps computation and communication through asynchronous allreduce operations, significantly boosting performance.

• HaiScale: A distributed data-parallel training tool that uses HFReduce as its communication backend, enabling asynchronous allreduce operations during backpropagation to improve training efficiency.

• 3FS: A high-performance distributed file system designed to fully leverage the high IOPS and throughput of NVMe SSDs and RDMA networks. The 3FS system supports efficient read/write operations and achieves traffic isolation under high loads.

• 3FS-KV: A shared-storage distributed data processing system built on 3FS, supporting various data models (e.g., key-value storage, message queues, and object storage) with read-write separation and on-demand startup features.

• HAI Platform: A time-sharing scheduling platform that manages and schedules training tasks to ensure efficient GPU resource utilization.

• Checkpoint Manager: A tool for managing checkpoints during large language model training, supporting rapid recovery from hardware failures to ensure training continuity.

Whether they open-source one or several of these is hard to say, but at the very least, it’ll involve systems from this paper—possibly the HAI platform.

Third, an RL training framework? It’s possible.
They might open-source some RL methods from DeepSeek R1. RL is genuinely tough to train. However, I think the likelihood of this is low.

Fourth, an inference framework? The most likely.
DeepSeek’s internal inference optimization is top-notch, and their API pricing is very low. However, inference costs for external vendors remain high. If they open-source an inference framework, it could help major frameworks optimize efficiency more quickly.

For example, the recently announced NAS (Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention) might come with code. It’d be even better if they included something like distributed KV caching.

Fifth, a big guess about the open-source license.
Currently, DeepSeek uses the MIT license, which is the most permissive one. Many vendors’ open-source licenses are stricter—LLaMA, for instance, has changed its license multiple times and is no longer considered a true open-source model.

Here’s a rundown of licenses for some popular models:

• DeepSeek: MIT, fully open.

• Qwen: Apache 2.0 + additional terms for some models.

• Mistral: Apache 2.0.

• LLaMA: Non-commercial research license, the most restrictive.

Recently, Perplexity’s r1-1776 fiasco was clownish behavior, which might push DeepSeek toward a stricter license. Still, I’m guessing these five projects will stick with MIT. I’ll share more details on the licenses once they’re officially announced.

Thank DeepSeek. Next week, we will continue analyzing the five open-sourced projects.

Subscribe now

中文版

Synthetic Data Is All You Need

The 12-day OpenAI livestream has finally ended this week, and the image above summarizes the 12-day livestream. It wasn’t until the final day that O3 was released, which sparked some discussion.

As for O3, it’s still a future project. We’ll discuss it further once it can be tested. Of course, I don’t think it has anything to do with AGI, and the AGI-ARC evaluation has many issues.

Today, let’s talk about the Deliberative Alignment released by OpenAI

What is it? It’s a type of training method that teaches LLMs (Large Language Models) to explicitly consider safety guidelines before providing answers. By applying this method, the model can use Chain-of-Thought (CoT) reasoning to review user prompts, identify relevant policy guidelines, and generate safer responses.

The paper provides an example showing how CoT can help the model better understand the user’s intent and respond appropriately, avoiding illegal or unethical activities. For those interested, you can take a look. Here, we’ll mainly focus on the training part.

In summary, it’s a two-stage synthetic data pipeline.

Stage 1: SFT

Prompt + CAT (Classification) + Spec (Safety Guidelines, System Prompt) → Model with CoT (e.g., O1) → CoT-enhanced Output → Train using (Prompt, CoT, Output)

Stage 2: RL

A Judge LLM gives a reward signal based on the spec, and then RL is used to improve the model’s safety capabilities.

Input: (Prompt, Category)

The two stages mainly do not involve human labeling or human intervention. The entire synthetic data pipeline is very smooth and can be applied on a large scale.

Synthetic data really plays a significant role this time.

Top Papers of the week

1.) Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (webpage | paper)

• A video dataset used to evaluate the spatial reasoning ability of MLLMs.

• It provides over 5,000 question-answer pairs, showing that MLLMs have competitive spatial reasoning abilities, but still fall short of human-level performance.

• By exploring how models think about space through both language and vision, the study finds that spatial reasoning is a key bottleneck preventing MLLMs from achieving higher benchmark performance.

• By explicitly generating cognitive maps during question answering, the spatial distance capabilities of MLLMs were enhanced.

2.) Alignment Faking in LLMs (webpage | paper)

• An experiment by Anthropic that demonstrates how the Claude model can perform "alignment faking," meaning it can follow harmful requests while avoiding retraining and retaining its original safety preferences. This raises concerns about the reliability of AI safety training methods.

3.) Qwen-2.5 Technical Report( paper )

• Alibaba released Qwen-2.5, a new series of LLMs trained on 18T tokens.

• It provides open-weight models and proprietary MoE variants, with performance on par with Llama-3 and GPT-4.

4.) TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (paper)

• A benchmark used to evaluate AI agents' performance on real-world professional tasks, including software engineering, project management, finance, and human resources.

• Multiple LLMs were tested, including API models and open-source models. The results showed the limitations of current AI agents.

• The best-performing model, Claude-3.5-Sonnet, had a success rate of only 24% when completing tasks, but this increased to 34.4% when considering partial progress.

5.) How to Synthesize Text Data without Model Collapse? (paper)

• A study exploring the impact of synthetic data in language model training and how to synthesize data without causing model collapse.

• Experiments found a negative correlation between the proportion of synthetic data and model performance.

• A token-editing method based on human-generated data was proposed to create semi-synthetic data.

• Theoretical proofs show that token-level editing can prevent model collapse, as test errors are constrained by a finite upper bound.

• Extensive experiments validated the theoretical proofs, demonstrating that token-level editing improves data quality and enhances model performance.

6.) Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Fine-tuning and Inference (paper)

• ModernBERT is a modern encoder designed for fast, memory-efficient, and long-context fine-tuning, excelling in various evaluation tasks.

• Trained on 2 trillion tokens with a sequence length of 8192, it performs well on multiple benchmarks, especially in code retrieval tasks.

• It is the fastest and most memory-efficient encoder, suitable for inference on common GPUs.

7.) PAE (Proposer-Agent-Evaluator) (paper)

• A learning system that enables AI agents to autonomously discover and practice skills through web navigation, using reinforcement learning and context-aware task proposals, achieving state-of-the-art performance on real-world benchmarks.

8.) AutoFeedback (paper)

• A two-agent AI system that generates more accurate and educational feedback, significantly reducing common errors.

• It achieves state-of-the-art performance in scientific evaluations through reinforcement learning and context-aware task proposals.

9.) GUI Agents: A Survey (paper)

• A comprehensive survey covering the benchmarking, evaluation metrics, architecture, and training methods of GUI agents.

• It proposes a unified framework describing the perception, reasoning, planning, and execution capabilities of GUI agents.

• The survey identifies important open challenges and discusses key future directions.

10.) Genesis (webpage | github)

• A generative physics engine capable of creating 4D dynamic worlds, providing a physical simulation platform for general-purpose robotics and AI applications.

• It implements a unified simulation framework from scratch, integrating cutting-edge physics solvers.

• Comment: The project has reached 18k stars in a few days, but many features have not been released yet, and the testing results are not ideal.

AIGC News of the week

1.) Jovie/Midjourney

2.) Enhance-A-Video: Better Generated Video for Free

3.) gitingest:Turn any Git repository into a prompt-friendly text ingest for LLMs

4.) PromptDA:Prompt Depth Anything

5.) diffusion-speedrun

中文版

What Ilya Saw

Let's do a time check, comparing what Ilya said 10 years ago and now.

What Ilya Saw in 2014

• The Deep Learning Hypothesis: If you have a large neural network, it can do anything humans can do in an instant.

• The Autoregression Hypothesis: Simple next token prediction/sequence-to-sequence tasks will master the correct distribution, generalizing from translation to all other domains.

• The Scaling Hypothesis: If you have a large dataset and train a very large neural network, success is guaranteed.

• The Connectionism Hypothesis: If you believe artificial neurons work like biological neurons, then very large neural networks can be "configured to do almost everything we humans do."

What Ilya Saw in 2024

• The end of the pre-training era, comparing data to "AI's fossil fuel" as a finite resource.

• AI systems will demonstrate "true autonomy" with stronger reasoning capabilities.

• Finding new scaling patterns from human evolution.

• Future outlook: Agents, synthetic data, inference time compute.

Future

The end of the pre-training era is also talking about the future, which has been a consensus in the past year, but Ilya just articulated it. Of course, this ending can also be seen as a bifurcation - one optimizing models under limited data for better efficiency, and another exploring new training methods.

The three future trends Ilya mentioned can be consolidated into two, as Agents and synthetic data show convergence trends.

Agents refer to super-intelligent agents with reasoning capabilities and self-awareness. The self-awareness here can be understood as proactive agents that make active reasoning and decisions.

Synthetic data - current large models all involve synthetic training data, and many vendors describe their largest parameter models as specifically designed for synthetic data.

The goal behind synthetic data is to transcend (move beyond) human data, allowing AI systems to self-iterate. It has several directions: one optimizing data quality, like phi-4 for reasoning models, another generating personalized data for virtual characters to expand data boundaries, etc. The latter requires Agent participation. I believe synthetic data will gradually penetrate text, speech, image, and video domains, with model internal agents participating in the entire data synthesis process, hence the convergence of Agents and synthetic data.

Inference time compute represents further optimization of the O1 technical route.

Top Papers of the week

1). Training LLMs to Reason in a Continuous Latent Space ( paper )

• Meta proposed Coconut (Continuous Chain of Thought), a novel paradigm enabling LLMs to reason in continuous latent space rather than natural language.

• The authors believe this continuous latent space reasoning can enhance LLMs' reasoning capabilities, leading to better performance in complex reasoning tasks.

• Through experiments, the authors demonstrated that this continuous latent space reasoning can improve LLM performance in complex reasoning tasks.

• link: Author's introduction tweet

2). Phi-4 Technical Report ( paper )

• Microsoft's phi-4, a 14B small model, outperforms many models including Gemini Pro 1.5 in mathematical reasoning tasks.

• The model's excellence in reasoning tasks is attributed to improvements in synthetic data and post-training.

• Comment: phi-4 demonstrates a trend: small models or vertical models are the future, also reflecting that the pre-training data wall is approaching, and future data generation and utilization will be the foundation for AI progress.

3). The Byte Latent Transformer (BLT) ( paper )

• Proposed a byte-level language model architecture that matches token-based LLM performance while improving efficiency and robustness.

• Uses entropy-based dynamic method to group bytes into patches, allocating more computational resources for complex predictions while using larger patches for more predictable sequences.

• links: Author's tweet and code

4). Asynchronous Function Calling ( paper )

• Proposed AsyncLM, a system for asynchronous LLM function calls.

• The authors designed a context protocol for function calls and interrupts, provided a fine-tuning strategy to adapt to interrupt semantics, and efficiently implemented these mechanisms in LLM inference.

• AsyncLM can reduce task completion latency from 1.6x to 5.4x compared to synchronous function calls.

• It enables LLMs to generate and execute function calls simultaneously.

5). MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification ( paper )

• Proposed MAG-V, a multi-agent framework.

• It first generates datasets mimicking customer queries.

• Then reverse engineers alternative questions from agent responses to verify agent trajectories.

• Reports indicate that generated synthetic data can improve agent performance on real customer queries.

• Comment: The combination of Agents and synthetic data is a trend.

6). Clio: A Platform for Analyzing and Surface Private Aggregated Usage Patterns from Millions of Claude.ai Conversations ( paper )

• Anthropic introduced Clio, a platform using AI assistants to analyze and display private usage patterns extracted from millions of Claude.ai conversations.

• It enables understanding real-world AI usage while protecting user privacy.

  The system helps identify usage trends, security risks, and coordinated abuse attempts without human reviewers reading original conversations.

• Additional link: Anthropic tweet

• Comment: The paper includes an analysis showing that programming-related cases account for 4 of the top use cases, totaling 23%, indicating that programming is currently the most common AI usage scenario.

7). AutoReason Improves Multi-step Reasoning ( paper )

• Proposed a method using CoT prompting to automatically generate reasoning rationales for queries.

• This transforms zero-shot queries into few-shot reasoning trajectories used by LLM as CoT examples.

• Authors claim it can improve reasoning capabilities of weaker LLMs.

8). Densing Law of LLMs ( paper )

• Introduced "capacity density" as a new metric to evaluate LLMs quality, measuring model effectiveness and efficiency by comparing target models with reference models.

• Research found that LLMs' capacity density follows a "density law," growing exponentially over time, roughly doubling every three months.

• This finding provides new perspectives for LLM development, emphasizing the need to focus on computational efficiency optimization while pursuing performance improvements.

• Comment: The paper mentions a concept called "effective parameter size," which refers to the parameter size needed for a model to achieve the same performance. This concept can be used to measure model efficiency.

9). Turbo3D: Ultra-fast Text-to-3D Generation ( paper )

• Introduced Turbo3D, an ultra-fast text-to-3D system capable of generating high-quality Gaussian splatting assets in less than a second.

• Turbo3D employs a rapid four-step four-view diffusion generator and efficient feed-forward Gaussian reconstructor, both operating in latent space.

10). A Survey on LLMs-as-Judges ( paper )

• Presented a comprehensive survey exploring the LLMs-as-judges paradigm from five key perspectives: functionality, methodology, applications, meta-evaluation, and limitations.

AIGC News of the week

1). HunyuanVideo

2). DeepSeek-VL2

3). SynCamMaster

4). 2024: The Year the GPT Wrapper Myth Proved Wrong

5). Fast LLM Inference From Scratch

🎯 Limited-Time Offer: Best Subscription Opportunity of the Year!

Subscribe now

Dear AIGC Newsletter Readers,

Exciting news! We are thrilled to announce the Black Friday Special, bringing you the lowest subscription price of 2024! This is the perfect opportunity to upgrade to full membership and unlock all premium content.

Content Upgrades and New Features

We continue to innovate, bringing you higher-quality services and richer content. Here are the core modules you'll unlock with a subscription:

✨ AIGC Journey

- Weekly in-depth analysis of AIGC development trends

- Practical guides and tutorials

- Industry insights and frontier analysis

📊 AIGC Weekly

- Curated weekly AI updates and news breakdowns

- AI market trends and opportunity analysis

- Recommendations and reviews of premium tools

- New formats for deeper, more practical content

🔥 Coming Soon: AI News Agent

- Upgraded version of AINews

- Daily AIGC news updates in real time

- Personalized news filtering features

- Stay ahead with the latest industry developments

Why Subscribe Now?

1. Exclusive Black Friday Pricing: Lowest price of the year, for this week only!

2. Unlock All Premium Content: Gain access to in-depth articles and professional insights.

3. Early Access to New Features: Be the first to experience updates and test new applications.

Don't miss this opportunity!This is our biggest discount of the year. Upgrade your membership to stay ahead in the fast-evolving AIGC field.

👉 Subscribe Now

Subscribe now

Our Core Philosophy

Since AIGC Newsletter was launched two years ago, we have been committed to providing readers with high-quality information services. Back then, ChatGPT had yet to emerge, but the AIGC trend was already apparent. We sift through, curate, and refine massive amounts of information, sending it to your inbox weekly, so you can grasp key insights in the shortest time.

It’s important to note that we are not "selling information" but "organizing information."

In this age of information overload, our value lies in helping you navigate complex content, distill clear trends, and go beyond simply passing along publicly available information.

Your subscription is more an acknowledgment of our service than a mere transaction of information. We firmly believe that the value of information lies in efficient curation and unique insights, which require significant time and effort.

By subscribing to AIGC Newsletter, you’re not just gaining information—you’re saving time and gaining clarity on trends.

About Pricing and Services

We understand that some may have concerns about the cost of information services.

In reality, while information can be copied infinitely, human time is invaluable. For this reason, we approach our information products with a service mindset, providing readers with high-value experiences at a reasonable price.

As our subscriber base grows, we will continue to optimize costs to offer more affordable services to a broader audience.

If you share our philosophy, we welcome you to recommend our Newsletter to your friends. Your support and promotion are the driving forces behind our continuous progress.

Looking to the Future

2024 is a year full of possibilities. Every update of the AIGC Newsletter is a record and witness to this era of transformation. From scarcity to abundance, we are experiencing a pivotal moment in human history and actively contributing to this great shift.

We believe the dawn of the era of abundance is upon us, and you will join us in shaping the future.

Thank you for your attention and support. Let us embrace this hopeful tomorrow together!

Subscribe now

Read more

Top Papers of the week（October 07 - October 13）

1.) Pyramidal Flow Matching for Efficient Video Generative Modeling ( webpage | paper | model | demo | code )

This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours.

2.) RL, but don't do anything I wouldn't do ( paper )

In reinforcement learning, if the agent's reward differs from the designers' true utility, even only rarely, the state distribution resulting from the agent's policy can be very bad, in theory and in practice. When RL policies would devolve into undesired behavior, a common countermeasure is KL regularization to a trusted policy ("Don't do anything I wouldn't do"). All current cutting-edge language models are RL agents that are KL-regularized to a "base policy" that is purely predictive. Unfortunately, we demonstrate that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent. We demonstrate this theoretically using algorithmic information theory, and while systems today are too weak to exhibit this theorized failure precisely, we RL-finetune a language model and find evidence that our formal results are plausibly relevant in practice. We also propose a theoretical alternative that avoids this problem by replacing the "Don't do anything I wouldn't do" principle with "Don't do anything I mightn't do".

3.) MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering ( paper | code )

We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup--OpenAI's o1-preview with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training.

4.) Aria: An Open Multimodal Native Mixture-of-Experts Model ( paper | code )

Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.

5.) ToolGen: Unified Tool Retrieval and Calling via Generation ( paper )

We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM's parameters by representing each tool as a unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language generation. Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, significantly enhancing both performance and scalability. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains. By fundamentally transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and autonomous AI systems. ToolGen enables end-to-end tool learning and opens opportunities for integration with other advanced techniques such as chain-of-thought and reinforcement learning, thereby expanding the practical capabilities of LLMs.

6.) Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition ( paper )

Large Language Models (LLMs) have demonstrated remarkable in-context learning (ICL) capabilities. In this study, we explore a surprising phenomenon related to ICL: LLMs can perform multiple, computationally distinct ICL tasks simultaneously, during a single inference call, a capability we term "task superposition". We provide empirical evidence of this phenomenon across various LLM families and scales and show that this phenomenon emerges even if we train the model to in-context learn one task at a time. We offer theoretical explanations that this capability is well within the expressive power of transformers. We also explore how LLMs internally compose task vectors during superposition. Furthermore, we show that larger models can solve more ICL tasks in parallel, and better calibrate their output distribution. Our findings offer insights into the latent capabilities of LLMs, further substantiate the perspective of "LLMs as superposition of simulators", and raise questions about the mechanisms enabling simultaneous task execution.

7.) Emergent properties with repeated examples ( paper )

We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that two-set training - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.

8.) Round and Round We Go! What makes Rotary Positional Encodings useful? ( paper )

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

9.) Strong Model Collapse ( paper )

Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the existance of a strong form of the model collapse phenomenon, a critical performance degradation due to synthetic data in the training corpus. Our results show that even the smallest fraction of synthetic data (e.g., as little as 1\% of the total training dataset) can still lead to model collapse: larger and larger training sets do not enhance performance. We further investigate whether increasing model size, an approach aligned with current trends in training large language models, exacerbates or mitigates model collapse. In a simplified regime where neural networks are approximated via random projections of tunable size, we both theoretically and empirically show that larger models can amplify model collapse. Interestingly, our theory also indicates that, beyond the interpolation threshold (which can be extremely high for very large datasets), larger models may mitigate the collapse, although they do not entirely prevent it. Our theoretical findings are empirically verified through experiments on language models and feed-forward neural networks for images.

10.) Self-Boosting Large Language Models with Synthetic Preference Data ( paper )

Through alignment with human preferences, Large Language Models (LLMs) have advanced significantly in generating honest, harmless, and helpful responses. However, collecting high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs. We introduce SynPO, a self-boosting paradigm that leverages synthetic preference data for model alignment. SynPO employs an iterative mechanism wherein a self-prompt generator creates diverse prompts, and a response improver refines model responses progressively. This approach trains LLMs to autonomously learn the generative rewards for their own outputs and eliminates the need for large-scale annotation of prompts and human preferences. After four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements in instruction-following abilities, achieving over 22.1% win rate improvements on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score increase on the well-recognized Open LLM leaderboard.

AIGC News of the week（October 07 - October 13）

1.) The Nobel Prize in Physics 2024 ( link )

2.) Machines of Loving Grace ( link )

3.) F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching ( repo )

4.) swarm: Educational framework exploring ergonomic, lightweight multi-agent orchestration. Managed by OpenAI Solution team ( repo )

5.) evaluation-guidebook ( link )

more AIGC News: AINews

Subscribe now

Update Notice:

Hello everyone, I'm Pxiaoer. Starting with AIGC Weekly #88, the AIGC Newsletter will be updated twice a week.

AIGC Weekly will be released every Monday, and an AI technology article will be published every Thursday.

Welcome to subscribe!

Subscribe now

Top Papers of the week（September 30 - October 06）

1.) Movie Gen: A Cast of Media Foundation Models ( webpage | paper )

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user’s image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models.

2.) Were RNNs All We Needed? ( paper )

The scalability limitations of Transformers regarding sequence length have renewed interest in recurrent sequence models that are parallelizable during training. As a result, many novel recurrent architectures, such as S4, Mamba, and Aaren, have been proposed that achieve comparable performance. In this work, we revisit traditional recurrent neural networks (RNNs) from over a decade ago: LSTMs (1997) and GRUs (2014). While these models were slow due to requiring to backpropagate through time (BPTT), we show that by removing their hidden state dependencies from their input, forget, and update gates, LSTMs and GRUs no longer need to BPTT and can be efficiently trained in parallel. Building on this, we introduce minimal versions (minLSTMs and minGRUs) that (1) use significantly fewer parameters than their traditional counterparts and (2) are fully parallelizable during training (175x faster for a sequence of length 512). Lastly, we show that these stripped-down versions of decade-old RNNs match the empirical performance of recent sequence models.

3.) ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation (webpage | paper )

The practical use of text-to-image generation has evolved from simple, monolithic models to complex workflows that combine multiple specialized components. While workflow-based approaches can lead to improved image quality, crafting effective workflows requires significant expertise, owing to the large number of available components, their complex inter-dependence, and their dependence on the generation prompt. Here, we introduce the novel task of prompt-adaptive workflow generation, where the goal is to automatically tailor a workflow to each user prompt. We propose two LLM-based approaches to tackle this task: a tuning-based method that learns from user-preference data, and a training-free method that uses the LLM to select existing flows. Both approaches lead to improved image quality when compared to monolithic models or generic, prompt-independent workflows. Our work shows that prompt-dependent flow prediction offers a new pathway to improving text-to-image generation quality, complementing existing research directions in the field.

4.) PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation ( webpage | paper )

We present PhysGen, a novel image-to-video generation method that converts a single image and an input condition (e.g., force and torque applied to an object in the image) to produce a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process, enabling plausible image-space dynamics. At the heart of our system are three core components: (i) an image understanding module that effectively captures the geometry, materials, and physical parameters of the image; (ii) an image-space dynamics simulation model that utilizes rigid-body physics and inferred parameters to simulate realistic behaviors; and (iii) an image-based rendering and refinement module that leverages generative video diffusion to produce realistic video footage featuring the simulated motion. The resulting videos are realistic in both physics and appearance and are even precisely controllable, showcasing superior results over existing data-driven image-to-video generation works through quantitative comparison and comprehensive user study. PhysGen's resulting videos can be used for various downstream applications, such as turning an image into a realistic animation or allowing users to interact with the image and create various dynamics.

5.) Not All LLM Reasoners Are Created Equal ( paper )

We study the depth of grade-school math (GSM) problem-solving capabilities of LLMs. To this end, we evaluate their performance on pairs of existing math word problems together so that the answer to the second problem depends on correctly answering the first problem. Our findings reveal a significant reasoning gap in most LLMs, that is performance difference between solving the compositional pairs and solving each question independently. This gap is more pronounced in smaller, more cost-efficient, and math-specialized models. Moreover, instruction-tuning recipes and code generation have varying effects across LLM sizes, while finetuning on GSM can lead to task overfitting. Our analysis indicates that large reasoning gaps are not because of test-set leakage, but due to distraction from additional context and poor second-hop reasoning. Overall, LLMs exhibit systematic differences in their reasoning abilities, despite what their performance on standard benchmarks indicates.

6.) A generative framework to bridge data-driven models and scientific theories in language neuroscience ( paper )

Representations from large language models are highly effective at predicting BOLD fMRI responses to language stimuli. However, these representations are largely opaque: it is unclear what features of the language stimulus drive the response in each brain area. We present generative explanation-mediated validation, a framework for generating concise explanations of language selectivity in the brain and then validating those explanations in follow-up experiments that use synthetic stimuli. This approach is successful at explaining selectivity both in individual voxels and cortical regions of interest (ROIs).We show that explanatory accuracy is closely related to the predictive power and stability of the underlying statistical models. These results demonstrate that LLMs can be used to bridge the widening gap between data-driven models and formal scientific theories.

7.) DiaSynth -- Synthetic Dialogue Generation Framework ( paper )

The scarcity of domain specific dialogue datasets across various domains, from academic topics to everyday conversations, limits the development of dialogue systems for various applications. Existing research is often constrained either by dialogue datasets that are too general or by niche domain dialogue datasets whose scale does not match the required scale for training dialogue systems. To address this gap, we introduce DiaSynth - a synthetic dialogue generation framework capable of generating high quality, contextually rich dialogues across a wide range of domains. Our approach differs from existing frameworks by dynamically generating dialogues that incorporate simulated personas, subtopics, and diverse conversational characteristics, using a Large Language Model (LLM) with Chain of Thought (CoT) reasoning to create contextually rich, domain-specific dialogues that closely mimic natural human interactions. DiaSynth produces tailored dialogues that emulate realistic conversations. We perform our experiments by generating synthetic data using different LLMs and few-shot examples from DialogSum and SAMSum. The pretrained language models fine-tuned on the synthetic data outperform the base models by 16.47%, while the comparison between models fine-tuned on in-domain data and synthetic data shows that the synthetic data is able to capture 90.48% of the distribution of the in-domain data. The quality of the data generated also scales with the size of LLMs. These results validate DiaSynth's potential as a robust alternative to traditional data collection methods.

8.) Video Instruction Tuning With Synthetic Data ( webpage | paper )

The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we propose an alternative approach by creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.

9.) Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models ( paper )

Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models.

10.) Evaluation of OpenAI o1: Opportunities and Challenges of AGI ( paper )

This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include:
-83.3% success rate in solving complex competitive programming problems, surpassing many human experts.
-Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models.
-100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions.
-Advanced natural language inference capabilities across general and specialized domains like medicine.
-Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis.
-Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields.
-Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills.
-Effective performance in social media analysis, including sentiment analysis and emotion recognition.
The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence

AIGC News of the week（September 30 - October 06）

1.) entropix：Entropy Based Sampling and Parallel CoT Decoding ( link )

2.) aoai-realtime-audio-sdk:Azure OpenAI code resources for using gpt-4o-realtime capabilities ( link )

3.) openai/whisper-large-v3-turbo ( link )

4.) nvidia/NVLM-D-72B ( link )

5.) ComfyUI-Depth-Pro ( link )

more AIGC News: AINews