X

Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet?

Finally met Leanna from @MiniMax_AI, and grateful for letting me early test the models! Eager about the next one! 👀👀

The latest 𝕏 algorithm has been published to GitHub github.com/xai-org/x-algorit…

GOOGLE 🔥: New Gemini Spark screenshots featuring advanced tool use and Skills creation flow. It seems like there won't be an option to import SKILL MD files besides copeing and pasting. There is also no evidence of Browser or Computer Use atm.

OpenAI is working on a dedicated setting for Codex to allow users to enable "Locked use." > Let Codex use your Mac while it's locked No more need to carry a half-open laptop around?

Minimax 🔥🔥🔥 shipping across modalities

You've been asking for this one... Now in preview: Codex in the ChatGPT mobile app. Start new work, review outputs, steer execution, and approve next steps, all from the ChatGPT mobile app. Codex will keep running on your laptop, Mac mini, or devbox. Video

Kill your SaaS with Zo Computer – Live from Singapore 🇸🇬 nitter.net/i/broadcasts/1qKVmQBbk…

A lot of the code shipped in App Store Connect CLI has been with @MiniMax_AI M2.5 recently and the screenshot automation feature (soon) is brainstormed with it I got early access to try it out, and I am experimenting with replacing Opus 4.6 *fast*

We've published a paper that explains our views on AI competition between the US and China. The US and democratic allies hold the lead in frontier AI today. Read more on what it’ll take to keep that lead: anthropic.com/research/2028-…

1/5 MiniCPM-V 4.6 (1.3B) is now live 🚀🚀 High-res visual processing, optimized for consumer-grade and mobile hardware. We’ve leveraged the latest LLaVA-UHD v4 technique to cut vision encoding costs by 55%, enabling native edge deployment with extreme efficiency. 🔥 Beats Gemma4-E2B-it and Qwen3.5-0.8B across key multimodal and Artificial Analysis benchmarks — scoring higher than Qwen3.5-0.8B using just 2.5% of its token budget. ⚡ TTFT (75.7ms) 2.2x Faster than Qwen3.5-0.8B even with 3136² high-res images. 🏗️ ~1.5x Token Throughput compared with Qwen3.5-0.8B on a single RTX 4090. Try the model here: 🤗 Hugging Face: huggingface.co/openbmb/MiniC… 💻 GitHub: github.com/OpenBMB/MiniCPM-V 🔭 Modelscope: modelscope.cn/models/OpenBMB… 🌐 Web Demo: huggingface.co/spaces/openbm… 📱 App Demo: github.com/OpenBMB/MiniCPM-V… Video

这个项目也可以直接 # 安装成 Claude Code skill 命令：violin --install-skill 以后就可以直接这样：violin input.mp4 output_zh.mp4 --language Chinese 大家需要注意： 去 api.together.ai 注册获取 Key（也支持 OpenAI、ElevenLabs，只需其中一个）。 Violin 默认使用 Together AI（免费注册可得额度），需要设置环境变量： # 永久生效，加到 ~/.zshrc echo 'export TOGETHER_API_KEY=你的key' >> ~/.zshrc source ~/.zshrc

Multi-agent's quiet problem: token cost scales with agents × turns × tool calls. It compounds fast , and that's the bill that kills production rollouts. @MiniMax_AI is now a native adapter in @JackChen_me's open-multi-agent, so goal-driven agent teams stay affordable in production, not just in demos. 👇 github.com/open-multi-agent/…