为什么是本地 AI，
为什么是现在？

大家能听到我们吗？测试一下声音？好的。请大家为 Local AI 鼓掌。呜！我希望你们和我们一样兴奋。这是——呜！这是本地AI峰会，所以我们今天一整天都会在这里讨论本地AI。原因是我们在今年达到了一个拐点。不仅模型变得非常好，而且工具链也变得非常好。而且这一切发生得非常快。我觉得在座的所有人想要跟上进度都很吃力。我最有这种感觉的时候是看到 Andre Karpathy 的一条推文。十一月，他发推说你还不能完全信任这些编码智能体。你得用鹰一样的眼睛盯着它们。三个月后，他发推说他都快跟不上这一切发展得有多好了，而事实是，他两次都是对的。这个领域进展非常快，你能做的事情多到令人难以置信。老实说，唯一能做的就是今天比昨天再多用一点点。这就是跟上这个领域的方法，而你们现在正在做的正是这件事。所以我对这场圆桌讨论非常兴奋。我们使用AI的方式也变了。我不只是在用聊天机器人，我不只是问简单的问题。当我们有了推理模型，

Can you guys hear us? Sound check? All right. Give it up for Local AI, everyone. Woo!I hope you guys are excited as we are. this is- Woo! This is the Local AI Summit, so we're gonna be here all day talking about local AI. And the reason why is we hit an inflection point this year. Not only did the models get really good, but the harnesses got really good. And this happened really fast. It's been, I think, a struggle for anyone here to keep up. I felt that most when I saw one of Andre Karpathy's tweets. In November, he tweeted that you can't really trust these coding agents alone yet. You have to monitor them with an eye like a hawk.Three months later, he tweets that he's struggling to keep up with the capabilities of how good this all has gone, and the thing is, both times he was right. This space is progressing really quickly, and it's wild how much you can do. And honestly, the only thing to do is just try to use it a little bit more today than you did yesterday. That's how to keep up with this space, and that's exactly what you guys are doing right here. And so I'm really excited about this panel. The way that we use AI has also changed. I'm not just using chatbots, I'm not just asking simple questions. When we got reasoning models,

AI模型回应的模式发生了变化。它不仅爆发式地回应我，而且在爆发之前，它会先进入一段平台期。它在推理，它在处理那些我不会消费的token。然后我们有了智能体，突然间我甚至不希望这些智能体关闭。我想要这些永远在线的智能体。只要我们设置好，它们就能一直保持高效。于是我们看到企业想把大量知识产权放进来，因为它变得更有用了。消费者也是一样。我想把我的健康数据、医疗记录给它。我想把我家用摄像头的录像给它。而无论是企业还是消费者，我们都不希望这些东西泄露。而且当你处于token持续生成的状态时，突然间成本就变得重要了。所以本地方案对这两点都非常出色。你可以确保在生成token时成本保持平稳，而且所有东西都留在那个房间里。所以在这几场演讲之后，我们有很棒的演示，

the, the profile of how the AI model responded changed. Not only is it bursty and responding to me, but before the burst, it kind of plateaus for a bit. It's reasoning, it's churning on tokens that I'm not consuming And then we got agents, and suddenly I don't even want these agents to turn off. I want these always on agents. They can always be productive if we set them up. And so we have enterprises that want to put a lot of their IP into this because it becomes more useful. We have consumerswith the same thing. I wanna give it my health data, my medical records. I wanna give it footage from my home camera. And both enterprises and consumers, we don't want that stuff to leak. And as you also have a profile of, tokens continuously generating, suddenly costs matter. So local is amazing for both of those things. You get to make sure that you are plateaued on the costs, for the tokens that you're generating, and also everything sits in that room. So we have amazing demos here after

所有运行的东西都留在那些设备上，留在这个房间里，所以这是一个非常好的保障。那么在我们进入圆桌讨论之前，首先，你们想自我介绍一下吗？

these talks where the, everything that's being run stays on those devices, it stays in this room, and so that's a really nice guarantee. So As we turn over to the panels, first, do you guys wanna introduce yourselves?

好，我先来？请。你有麦克风吗？有，你的麦克风就是那个麦克风。所以，嗯，我是 Alex。我是 Exo Labs 的联合创始人兼CEO，也是 Local dot AI 的创建者。我们已经在本地AI上工作了两年多，在这个领域感觉要长得多，而且，你知道，从在两台 MacBook 上运行 Llama 405B 的日子，到现在我们有——你知道，那边的演示——Nimitron Ultra 在四台 Sparks 上运行。你知道，我们的使命是让AI更容易被使用，而且，你知道，看到现在可能做到的事情真的很疯狂。太酷了。

Yeah, should I start? Please. You got the mic? Yeah, your mic's the mic. So, yeah, I'm Alex. I'm the co-founder and CEO of Exo Labs and also the creator of Local dot AI. So we've been working on- Local AI for over two years, which feels like a lot longer in this space, and, you know, from the days of, you know, running llama 405B on two MacBooks to now where we have, you know, Demo there, Nimitron Ultra running on four Sparks. You know, our mission is to make AI more accessible,and, you know, it's crazy to see what is possible now.cool.

大家好，我是 Matt。我做内容、视频，我们有一份关于人工智能的新闻通讯。我是个AI爱好者，是的，谢谢邀请我。

Hey, everybody, I'm Matt. I make content, videos, we have a newsletter all about artificial intelligence. I'm an AI enthusiast, and yeah, thanks for having me

大家好，我叫 Ahmed Osman，我是 OsmanTech 的创始人兼CEO。有人私信问我："嘿，OS Man 代表什么？" Open Source Man（开源之人），所以那就变成了一家公司。而且，我也是 local lambda subreddit 的主持人。我从2022年就开始涉足本地AI领域了，而且，是的，让我们把开源和本地AI合而为一。我很喜欢。

Hey everyone, my name is Ahmed Osman, I am the founder and CEO of OsmanTech. somebody jumped into my DMs asked, "Hey, what does OS Man stand for?" Open Source Man, so that became a company. And, I also moderate local lambda subreddit. I have been in this local AI space since twenty twenty-two, and, yeah, let's, let's make open source and local AI one. I love it

开源之人得到了大家的掌声。这很合适。今天，我觉得，对这场圆桌讨论来说是个相当重要的日子。你知道，Fable 刚回归，但这很好地提醒了我们，为什么我们需要获得前沿智能的访问权限才能构建任何东西。我是 Joseph，我是 Roboflow 的联合创始人兼CEO。我们做所有与视觉相关的事情。我有点喜欢开玩笑说，视觉就像是最初的本地AI，因为所有东西可能都需要在图像和数据被捕获的同时，与你的视频一起以低延迟并发运行。期待今天的讨论，以及所有关于为什么本地AI是让AI对每个人都有用的未来的话题。

Open source man gets the round of applause.That's appropriate. Today's, I think, a pretty momentous day for the panel. you know, Fable just came back, but it's a good reminder of why we need to have access to frontier intelligence to be able to build anything. I'm Joseph, I'm co-founder, CEO at Roboflow. We do all things vision. I kinda like to joke that Vision is like the original local AI, 'cause like everything needs to run probably concurrent with like low latency alongside your video, where the images and data is being captured. looking forward to the discussion today and all things why local AI is the futureof making AI useful for everyone.

大家，让我们为Joseph和在座各位鼓掌。呜！那么，对于所有小组成员，我真的很兴奋的一点是，你们都以自己的方式很早就进入了这个领域，而且，你知道，非常早，我想我们都感受到了这个转折点——那么，你们是什么时候感觉到这个转折点的？

Everyone, let's give Joseph and everyone a round of applause. Woo!So, one thing that I'm really excited about for all the panelists is you guys were all early to the space in your own way, and, you know, very early, and I think as we all feel this inflection point- Now, when do you guys feel like you felt the inflection point?

是啊，我，我可以先来，当然。我的意思是，看，当我第一次看到Llama的时候，对吧？我非常兴奋地想到我可以真正下载这个智能，在我的本地电脑上运行它。这非常让人兴奋。我本质上是个喜欢鼓捣的人，是个喜欢构建的人。我——我多年来一直在超频PC，所以这感觉太好了。能够拥有这个疯狂的、外星般的智能在我的办公室设备上运行，感觉真的非常有趣。而且那，那是我第一次，说实话，我甚至没想到这是可能的，直到我看到了Llama，然后我想，“哦！哇，这太不可思议了。”所以那真的是我被它深深吸引的时刻。

Yeah, I, I can go first, sure. I mean, look, when I first saw Llama, right? I got really excited the idea that I can actually download this intelligence, run it from my local computer. It was very exciting. I'm a tinkerer at heart, I'm a builder at heart. I- I've overclocked PCs for years, so this just felt like so good. It just felt so interesting to be able to have this crazy alien intelligence running on, on my device in my office. and that, that was the first time where I didn't even think it was possible to be honest, untilI saw Llama, and I was like, "Oh! Wow, this is incredible. so that was really the point at which I was very turned on to it.

我，我想，对我来说也差不多。是的，是la matu，我想，终于在我的1490 RTX 1490上，我就像，哇，我实际上可以理解这个黑箱，定制它，调整参数，比如，你知道，采样参数以及所有那些配置，然后看看推理引擎是如何工作的。那，那，那让我可以控制它如何，如何配合我的思维方式。然后从那以后，你知道，剩下的就是历史了。基本上，我一直非常积极地表达为什么本地和开源AI必须与云共存，以及我们应当如何，你知道，去推动这一点，尽可能多地理解它，并教给人们。

I, I think, it's kind of the same for me. yeah, it's la matu, I think, that, finally on my 1490 RTX 1490, I'm like, wow, I can actually understand this black box and customize it and play with parameters and like, you know, sampling parameters and all those, configurations and see how inference engines work. that, that, that made I can't control how it, how it plays to my ways of thinking. and fromthere it was, you know, the rest is history. Basically, I've been very vocal about why local and open source AI must coexist with cloud and how, how we are supposed to, like, you know, push for that and try to understand it as much as possible and teach people about it.

完全同意。我实际上也感受到了这一点。我在一架飞机上，我想那是在2023年，我有一个模型在我的手机上运行，它糟糕透了。但当时我，我没有网络，所以我能够，比如，在飞机上问一些智能的东西，那是我第一次感觉，“嗯，这会是一件非常酷的事情。”

Totally. I actually felt this as well. I was on a plane, I think it was in twenty twenty-three, and I had, a model running on my phone, and it was awful. But I was, I didn't have internet, and so I was able to, like, ask, essentially intelligence something there in the plane, and that was my first feeling of like, "Hm, this is gonna be something really cool."

你知道吗，你现在可以在你的iPhone上运行相当于GT4O的模型？

Do you know that you can run the equivalent of GT4O on your iPhone now?

真的吗？是的，它是Quan

Really? Yeah, it's Quan

三点五，四个参数，四十亿参数。它，它基本上和以前在数据中心中提供的模型质量相同。而它就在你口袋里的设备中。

three point five, four parameters, four billion parameters. It, it's basically the same quality as something that used to be served in data centers. And it's in a device in your pocket.

是的，那太疯狂了。是的，我想如果你想想GPT-4o那个时刻有多疯狂，而现在你可以把它运行在手机上，这就像是，我想很多这些东西只是，你知道，你必须看到事物发展方向的愿景，因为它当时只是个玩具，对吧？就像是，你知道，第一——我想对我来说有几个关键时刻。一个是本地运行Llama 4.5，对吧？那就像是差距，它显著缩小了与前沿闭源模型和开源的差距，但它以每秒两个token的速度运行，所以没什么用，对吧？然后，你知道，另一个重要时刻是DeepSeek，V3和R1都是，DeepSeek V3是一个巨大的MoE，所以，你知道，Llama-405B是稠密的，所以超级慢。MoE似乎解锁了性能，所以就像是，哦哇，你知道，用我已经有的设备，比如，你知道，Mac，Mac Studio或Spark，我可以运行，你知道，这个巨大的模型，而且实际性能还不错，

Yeah, that's crazy. Yeah, I think if you just think about like how crazy that GPT-4o moment was, and now you can run it on the phone, it's like, I think a lot of this stuff was just, you know, you had to see the vision of like where things were going because it was a toy, right? It was like, you know, the, the first-- I think there were a few key points for me. One was like running Llama 4.5 locally, right? That was like the The gap, it closed the gap quite significantly with the frontier closed modelsand open, but it ran at two tokens per second, so it wasn't useful, right? and then, you know, another big moment was DeepSeek, Both V3 and R1, DeepSeek V3 was like a massive MoE, so like, you know, llama-405B was dense, so it was super slow. MoE seemed to unlock the performance, so it was like, oh wow, you know With the devices I already have, like with a, you know, a Mac, Mac Studio or Spark, I can run, you know, this massive model at actually like decent performance,

这与云端的运行能力相当，然后，我想就在最近，对我来说，GLM 5.2也是一个重要时刻，因为它再次缩小了差距，而且你知道，是Opus级别的，而且，你知道，我们已经让它在那边的一台设备上运行，它真的可以放在你的桌子上，就是DDX station。所以对我来说，这就像一个趋势，而且它会持续下去，对吧？比如会有越来越小的设备，内存更少，压缩更好，我们将能够在本地运行更有能力的模型，而且，你知道，很快这将成为默认。

which is comparable with, you know, what you can run in the cloud, and then, I think just recently, to me, GLM five point two is a big moment as well, 'cause again, it's closing the gap and it's like, you know, Opus level, and, you know, we have it running over there on a device that like literally can fit on your desk the DDX station. So this is, to me, just like, you know, a, a trend, and it's gonna keep increasing, right? Like there's gonna be smaller and smaller devices with less memory, bettercompression, we'll be able to run more capable models locally And, you know, soon it'll be the default.

我会继续这个，这个，飞机故事的主题。我，我同意在过去有过很多这样的时刻，关于发生了什么，但有一个——我有

I'll keep with the, the theme of, of airplane stories. I, I agree that there's been like multiple moments over- Over time of what's been, what's been going on, but one-- I have

在那之后我会有一个相关的演示，所以请留下来。

a presentation for that after this, so stay around, please.

好的。举个例子，这件事发生在我坐飞机的时候，我坐在一个人旁边——大概是两三年前——他们有视力障碍，他们当时在用，如果你用苹果手机，你知道，你给什么东西拍张照，可以用辅助功能设置，它会向你描述这张照片。我们一起坐在飞机上，他们不停地拍照，想搞清楚我们坐在哪里、怎么系安全带之类的事情。我记得他们拍了一张前面座椅靠背的照片，苹果的辅助功能描述了这张照片，把它描述得像一台打印机。当然，那个人根据上下文知道这不可能是对的，然后我想，你知道吗，我在想，Lava 刚刚出来，是个多模态模型，有点像转向以文本为基础来描述事物，而不是原生多模态，但还算是一个有用的例子。我很好奇，我就想，“好吧，看看 Lava 处理同样的事情会怎么样。”于是我拍了一张我面前座椅托盘的照片，它很恰当地描述了它。它说，“嘿，你在飞机上，这是座椅托盘。”然后我把它展示给坐在旁边的人看，那个人从未见过本地模型，也从未体验过本地 AI，让我印象最深的是，我想，“天哪，一家公司——”

All right. one example moment where this like came up for me was, I was on a plane, I was sitting next to someone Man, this is maybe two, three years ago, and they were hard of sight, and they were using, if you, on Apple, you know, if you take a photo of something, you can use the accessibility settings, and it'll describe the photo to you. And we were seated on the plane together, and they were taking constant photos to understand where we're seated and how to get buckled in and these sorts of things. And I remember they took a photo of the seat back in front of them, and the Apple accessibility describe the photo, describedit aslike a printer Of course, the individual knew from the context that couldn't be right, and I was like, you know, I wonder, Lava had just come out, multimodal model that like, kind of just described things with like the pivot to text base rather than like natively multimodal, but still kind of a useful example And I was curious, I was like, "Well, let's see how Lava would do on the same thing." So I took a photo of like the seat tray in front of me, and it aptly described it. It's like, "Hey, you're on an airplane, that's the seat tray." And I showed the person sitting next to me, who had never seen local models and never experienced local AI, and what that Stood out to me is I was like, "Man, a company that's

一家市值超过万亿美元的公司，在手机上推出用于描述视觉内容的最新智能技术，却不如一个任何人都能广泛获取和使用的东西。那真是一个分水岭式的、非常清晰的时刻，表明即使是

a trillion dollar plus market cap business shipping the latest intelligence on their phones for describing visual settings is inferior to something that's broadly accessible and available to anyone." And that was such like a watershed, clear moment that like even the

最大的公司在智能前沿领域也没有垄断权，所以，让这些东西越来越能被其他人获取，我认为，对于理解这项技术将产生的所有巨大影响至关重要。那就是 Lava。是的，很有意思，听你

Largest companies don't have a monopoly on the frontier of intelligence, and so increasingly making that be accessible to others, is, I think, gonna be critical for understanding all the massive impacts the tech will have. And that was lava. Yeah, it's interesting to

这么说——那大概是2022年、2023年。

hear you-- That's like twenty twenty two, twenty twenty three

差不多。对对对。是的，那是挺久以前的事了。听你讲到那种可获取性，很有意思，对吧？这种前沿智能，让它变得有用的是给它配上外设——在这个例子里就是摄像头——这样它就能获取你面前的数据。我认为这就是今年的拐点所在，远不只是模型本身，还有这些框架，以及你能融入智能体里的东西。那真是——你知道，GPT-4o 真的很令人兴奋，因为我当时用它来写代码，对吧？我从代码库里拿出代码片段，然后去 ChatGPT 把代码粘贴进去，我会给它我认为应该给的尽可能多的上下文，然后我把结果拿回来，回到代码库里粘贴进去。而像 Cursor 这样的工具真正了不起的地方在于，它本质上就是这个框架，它说：“如果我直接拥有完整的文件系统，让这些智能体自己去推理需要哪些文件，会怎么样？”所有这些本地模型和系统，关键问题是它如何——它如何与真实世界互动？我很好奇，Joseph，你们是从 Vision 起步的，看起来 Vision 不得不通过艰难的方式学到很多 LLM 现在正在处理、正在发现的东西。你觉得，语言领域现在正在发现的、我们可以从 Vision 那里借鉴的一个重要教训是什么？

around there. Yeah, yeah, yeah. Yeah, that was a while ago. It's interesting to hear you say about the, the accessibility of that, right? The, taking this frontier intelligence, what made it useful was giving it peripherals, in this case, camera, so that it could access the data in front of you. And I think that's where the inflection point this year was so much more than just models, but also these harnesses and what you canInto your agent, it was, you know, four O was really exciting because I was coding with it, right? I was taking code snippets from my code base, and then I'd go to ChatGPT and I'd paste it, I'd give it as much context as I thought I should, and then I would take the result and I'd go back into my code base and paste it. And what was really great about tools like Cursor was it essentially was this harness that said, "What if I just had the full file system, and I can let these agents reason on what files they neededAll of these local models and systems is how does it, how can it interact with the real world? I'm curious, Joseph, so you guys got the start in Vision, and it seems like Vision had to learn the hard way what a lot of LLMs are dealing, discovering right now. What do you think is like a big lesson that the language world is discovering right now that, we could look to Vision for?

我认为，语言的优势之一在于它本质上是人类构建的产物。所以语言通常存在于有人类的地方，这也意味着你通常可以使用数据中心里无限的算力资源，而很多视觉任务则受限于算力。你可能运行在网络连接较差的场景，运行在设备上，运行在机器人上，你能获得的算力就是那么多。那这带来了什么影响？我认为，这促使人们更重视专用模型，因为你会想，好吧，我需要这个模型在有限的条件下运行。我不会优先追求全世界的通用性，而是要专注于特定领域场景的工作。有趣的是，我认为你实际上正看到语言模型也在走类似的路。你是从特定场景的工具框架角度来描述的，但你知道，像任何类型的编程代理这样的工具，它们有很好的框架，专门用于处理这类任务，但你也会看到，即使在税务准备或法律准备这样的领域也是如此——

I think, one of the things that language has an advantage of is it's inherently a human construct. So language generally exists where people exist, and that also means usually you can use infinite amounts of, of computer, what is available in a, in a data center, whereas a lot of vision, you're compute constrained. You're running maybe where there's low internet connectivity, you're running on a device, you're running on a- A robot, and the amount of compute you have available to you is, is what, is what you get. And so what did that do? That created, I think, an emphasis on specializedlearners faster, because it's like, okay, I need this model to run in a limited context. And I'm not gonna prioritize full world scale generalizability, instead I'm gonna have specialized in domain context work. And what's interesting is I think you're actually seeing language follow a similar thing. You're describing it In the context of like harnesses for a given context, but you know, tools like, any sort of coding agent where you have a really good harness and is specialized for doing those sorts of tasks, but you're also seeing that even in like, you know, Tax preparation or legal preparation

——最后一公里的微调、适配、专用模型。所以在某些方面，我觉得钟摆正在摆回真正拥有专用模型的方向，即使在语言领域也是如此。视觉领域是一回事，然后，你知道，当你在小众设备上运行时，如何最大化算力优化则是另一回事，但我知道 Exo 的人肯定能谈这个。当然。但没错，我认为专用模型——还记得那种"一个模型将统治整个世界"的想法吗，而现在我觉得钟摆已经摆回来了，人们开始意识到

of last mile, fine tuning, adaptation, specialized learners. And so in some ways, I feel like the pendulum's swinging back to actually having specialized models, even in language. As much as Vision, I think is one thing, and then, you know, there's a whole different thing around how you get the most out of compute optimization when you're running on, on a niche device, but I know that probably the Exo folks can talk Absolutely. but yeah, I think specialized models being, remember there was like one model will rule the mall world, was the kind of thought, andI feel like the pendulum has swung back to people realizing

专用模型。我认为我们在英伟达确实看到，这将是一个多模型共存的世界。

specialized models I think we definitely, at Nvidia, we see that it's gonna be a multi-model world.

是的。

Yeah.

我们完全同意这一点。实际上稍后有一个小组讨论，我想是在下午三点二十分，主题就是模型路由。这真的很令人兴奋。我想作为一个爱好者，当你在使用这些工具时，你的使用模式是什么样的？你会使用很多模型吗？那大概是什么样的情况？

We definitely agree with that. there's actually one of the panels that's coming on later, I think at three twenty PM, is all about model routing. And so that's, that's really exciting. I guess as an enthusiast, and as you use some of this, is that, How your usage pattern looks? Are you using many models? What does that kind of look like?

是的，当然。从明显是顶级模型的 Fable，到像 Sonnet 这样更务实的主力模型，再到本地模型——用于那些不太依赖低延迟的任务。是的，多模态世界在这一点上似乎已经是显而易见的了，尤其是当你看到所有这些企业公司审视自己的预算，想着，好吧，作为一家公司我需要继续增加消耗的 token 总量，但也不想把预算完全花光。我知道 Coinbase 和 Brian Armstrong 前几天刚发了一篇很棒的文章，谈到他们的 token 消耗量在爆炸式增长，但成本却保持平稳，那是因为他们使用了不同模型的组合。你不需要在每个用例中都使用顶级模型，事实上大多数用例都不需要。我认为最明显的应用方式是让顶级模型来规划架构，无论顶层计划是什么，然后实际的代码执行则交给——

Yeah, absolutely. Everything from obviously like the top models, Fable, to kind of the more workhorse models with Sonnet, local models for things that maybe don't depend as much on low latency yeah, like the multimodal world s-seems like a no-brainer at this point, especially as you're seeing all of these enterprise companies come out and look at their budgets and, and think, okay, well, I, I need to continue to increase the, the total number of tokens that I'm consuming as a company, but Ialso don't wanna just completely blow out my budget. I know Coinbase and Brian Armstrong just came out with that just great post the other day talking about, How their tokens are, are exploding, yet their costs are staying flat, and, and that is because they're using a mixture of different models. You don't need the top model for every single use case, and in fact, most use cases you don't. I think the most obvious application is let the top model plan, the, the, the architecture, whatever the kind of top level plan is, and then the actual execution of

——代码执行可以交给一个价格更合理的小型模型。

the code can go to, a more reasonably priced smaller model.

你最智能的模型应该为你提供整体规划，然后子任务交给更小的执行模型——就像执行者模型，这正是未来。

Your most intelligent should provide you with the overall plan and then subtasks For your smaller execution, like e-executioner models, and that's exactly the future.

是的，尤其是，你知道，我们说到本地模型。本地模型很擅长写代码，但也许我们把实际的顶层规划交给一个前沿模型，这样你能省下不少钱，对工作流有更多掌控，这是一个很好的模式。

Yeah, especially like, you know, we're talking about local, The local models are great at writing code, but maybe we offload the actual top level planning to one of the frontier models, and you, you save a bunch, you control more of the workflow, it's, it's a, it's a nice pattern

是的，我，我认为市场想要这个。所以几年前当这些东西刚开始发展的时候还不清楚，你知道，会不会只有一个大模型被所有人使用？但显然那不是人们想要的，那不是企业想要的。他们不想被Dario告诉他们能做什么。他们不想为所有工作负载支付同一个模型的费用，而有些工作负载其实并不需要，你知道，一个巨大的、巨型模型，每百万token要花五十美元。他们想要控制权，他们想要主权，他们想要能够切换模型的能力，他们不想一夜之间被人釜底抽薪，你知道，因为某些安全风险或者其他什么原因。而且，你知道，这确实是推动很多进展的原因。所以，你知道，我比以往任何时候都更乐观，我觉得，关于本地化的走向。我认为，你知道，市场基本上是在把这些东西从创业公司里拉出来，你知道，它，

Yeah, I, I think the market wants this. So it wasn't clear like a few years ago when this stuff was just starting to play out, you know, would there just be one big model that everyone is using? But clearly that isn't what people want, that isn't what enterprises want. They don't want to be told what they can do by Dario. They don't want to be paying for the same model for all their workloads when some workloads don't actually need, you know, a giant, gigantic model that costs fifty dollars per million tokens. They wantcontrol, they want sovereignty, they want the ability to switch out models, they don't want to get rug pulled, you know, from one day to the other, because of some safety risk or whatever. and, you know, that's really what's driving a lot of this progress. So, you know, I'm more optimistic than ever, I think, about sort of, you know, where things are going with local. I think that, you know, the market is basically pulling a lot of this stuff, out of, you know, startups, out You know, it,

它，它，是的，它，它正在发生，对吧？

it's, it's, yeah, it's, it's happening, right?

完全同意。我觉得这是一个新前沿，当我们进入多模态时，那变得非常困难。当然，你需要在模型之间进行路由，但随后你必须弄清楚如何为你路由到的任何模型提供必要的上下文。如果你在制定一个计划然后把它拆分成子代理，那用什么框架来做？你应该怎么做？这些都是开放性问题，而且，你知道，我认为仅仅是玩一玩，市场就是在推动这些答案。它正朝这个方向延伸，我们在寻找创业公司和公司来填补这个空白，这非常有道理。你知道什么好笑吗？当任何企业消费任何软件的任何一部分时，如果你要建立一个基础，你很想了解版本是什么，你很想了解你的模型是什么，哪个，那个版本是否改变了，以便之后看到行为的变化。所以，对所有这一切有更多控制权，这样你就确切知道它是什么版本，这甚至不是因为任何，你知道，监管，而仅仅是因为如果有更新，你可以选择什么时候在你整个技术栈中决定采纳任何更新。

Totally. I feel like this is a new frontier where as we go multimodal, that becomes really difficult. Of, of course, you need to route between the models, but then you have to figure out how to provide necessary context to whichever model you're routing to. If you are making a plan and then breaking it off into sub-agents, what is the framework to do so? How should you do this? These are all open problems, and, you know, I think just being playfulis,the market's kind of pulling for these answers. It's reaching in this direction, we're looking for startups and companies to fill this gap, makes a ton of sense. You know what's funny? as ent- any enterprise consumes any s-any piece of software, it matters a lot if you're gonna build a foundation, you wanna know what the versions are, you wanna know what your model is, which, whether that version changed in order to see changed behavior later. So having, also more control over all of this so that you know exactly what version it is, it Not even because of any sort of, you know,regulation, but just simply if there are updates, you choose when you're opting into whatever it is on your entire stack.

是的。那么，当你思考我们在这里看到的这股浪潮，这个被称为AI的文明基础设施时，你必须考虑，你知道，东西可能被从你手中夺走以及你的主权的潜（在风险），以及你如何能够负责，你知道，端到端地测试整个技术栈？那是硬件、软件以及介于两者之间的所有东西。那是模型权重，那是模型的特定版本，正如你所说的，那是你如何微调它的方式。还有，接着Joseph的，你知道，观点，关于小型和专业化模型的陈述，我一直以来，你知道，我一直是这方面的支持者有一段时间了。我在2024年发过推文说小型和专业化模型是未来，而且它将是为每个用户、每个用途、每个使用场景、每个企业的工作流，你知道，来决定他们的细分领域，你现在就需要开始接纳它们，因为你想要收集那些竞赛的数据点。那就是

Yeah. Now, so When you think about,this wave that we're seeing here of like this civilizational infrastructure that is called AI, you have to Consider the, the potential of, you know, things being taken away from you and your sovereignty, and, how can you be in charge of, you know, testing full stack end to end? That's hardware, software, and everything in between. That's the model weights, that's the specific version of the model, as you were saying, that's how you fine tuneit. And picking up, Joseph's, you know, thing, statement here about small and specialized models, I have been Like, you know, I have been a proponent of that for quite some time. I have tweets about that in twenty twenty four saying small and specialized models are the future, and it is gonna be per user, per use, per use case, per workflow for businesses, you know, to, to decide on their niche domains, and you need to start onboarding them now because you wanna collect the data points that races. That's what

你在问的，Nader。你在问我们怎么到达那里？我们如何决定哪个模型被路由到？我们如何决定哪个用途去哪个模型，或者哪个端点？你需要收集数据，并且你需要准备好基本上拆解你从你的员工、从你组织中的每个人那里收集到的轨迹，然后决定我们如何能最优化地利用这些数据点来对应哪些模型，并且也收集反馈。而且你实际上也可以用代理来自动化这个过程。所以这就像是，你知道，关于RSI（递归自我改进）的整个事情，它也适用于甚至代理和框架，也适用于工作流和使用场景以及企业。你想要向所有人描述一下

you're asking about, Nader. You're asking about how do we get there? How do we decide which ma-model gets routed to? How do we decide which use Gets, goes to which model or which, which endpoint? You need to collect data, and you need to be ready to basically break down, the, the traces that you've collected from your employees, from everybody in your organization, and decide how can we make the most optimal use case of these data points to which models and collect, collect feedback as well. And you can actuallyautomate that with agents as well. So like, this is like something that it's, you know, the, the whole thing about RSI right now, recursive self-improvement Improvement. It also applies to even agents and harnesses and to, to workflows and to use cases and to enterprises. You wanna describe that

给每个人，也许给那些不知道RSI的人？

for everyone too, maybe for folks who don't know RSI?

递归自我改进。这意味着，一个模型基本上会租用自己的算力，然后开始训练自己的检查点，然后，你知道的，部署它的下一个版本，以便在某些方面得到更新，在某些方面改变行为。所以基本上，一个模型在训练自己。

a recursive self-improvement. It means that, a model would basically rent its own compute and then start training its own checkpoints and then, you know, deploy its next version so that it gets updated in certain ways, changes behavior in certain ways. So basically, a model is training itself.

完全同意。我们的产品 Brev，它让获取 GPU 变得非常简单，我们一直更多地关注智能体作为第一类受众，因为我们看到了这一点。所以我们看到越来越多的智能体想要直接去获取 GPU。

Totally. Our product, Brev, which just makes it really easy to get a GPU, we've been focusing more on agents as a first-class audience because we're seeing this. So we're seeing growing usage from agents that just want to go grab a GPU directly.

正是如此。

Exactly.

所以，是的。你知道吗，有趣的是，你提到的第二件事，不仅仅是多模型，还有所有这些优化，以便在你拥有的算力少于数据中心的情况下真正运行它、执行它。我觉得 Alex，这是我们做过的一些有趣的工作。所以，Alex 实际上在 Nvidia 内部设立了一个第二总部。好吧。我们有一间会议室，然后——我们当时在吃晚饭，他提到他确实有动力去榨干我们在 DGX Spark 上能获得的每一滴性能。所以我们说，嘿，让我们在 Nvidia 弄一间会议室。把你的团队带来——

So, yeah. You know, it's funny, so the second thing you talked about was, not just multi-model, but then all these optimizations to actually run it, perform it when you have less compute than, say, data center. I think Alex, this is some fun work that we did. So, Alex has actually set up, a second headquarters inside of Nvidia. Okay. We got a conference room and- we were, we were having dinner, and he mentioned that he really was motivated to get, to squeeze out every drop of performance we could on the DGX Spark. And so we said, hey, let's get a conference room at Nvidia. Come bring-

你的团队，而且任何时候你需要在任何领域有专家，我们就去把那个人拉进会议室。

Your team, and anytime you need an expert across any pillar, let's just go and pull that person into the room.

而且，是的，我，我实际上有一个后续问题。所以既然我加入 Local Llama 已经有一段时间了，当我想到家庭实验室爱好者时，我想到的是那些实际上在企业内部、在董事会会议室里做决策的人，你知道，关于我们要托管哪些模型，或者你知道，我们要从哪里获取 AI。这些家庭实验室关注的事情是优化。基本上就是，我如何从硬件、我拥有的限制、我拥有的软件中提取最具经济价值的产出。你知道，量化就是这样出现的。当我们考虑企业时，也是一样的。是限制、是预算限制、是我如何在我拥有的硬件下最大化利用我能获得的 token 数量，无论是 DGX 工作站那种规模的企业，还是你知道，DGP 三百集群。这一切都是关于，我如何为最新最前沿的开源模型优化软件，并在我们拥有的经济限制下从中获得尽可能多的价值。我认为看到家庭实验室爱好者至关重要，这就是为什么，你知道，这是本地 AI。它是本地 AI，但它可以——它可以在本地部署，它可以是——托管硬件，它可以是租用的集群，它可以是来自 Brev 的，是的。基本上，这就是关于

And, yeah, I, I actually have a follow-up here. so since I have been part of Local Llama for quite some time, when I think about home labbers, I think about the individuals that are actually, are within enterprises, in boardrooms, making decisions about, you know, the models that we're gonna host or You know, where we're gonna get our AI from. the, the things that these home labs focus on are optimizations. It's basically how can I extract the most economical value out of the hardware, the constraints that I have, the software that I have.You know, quantizations came to be because of that. When we're thinking about enterprises, it's the same thing. It's constraints, it's budget limitations, it's how can I make the most use of the number of tokens that I can get under the hardware that I have, whether that's a DGX station, you know,sized business or, you know, a DGP three hundred cluster. It's all about how can I optimize the software for the latest and greatest frontier open source model out there and get as much value as possible out of it under the economical constraints that we have. I think it's, it's crucial to see home labors, and that's why, you know, this is local AI. It's local AI, but it can-- it could be on premises, it could be- Colocated hardware, it could be rented clusters, it could be from Brev, yeah. Basically, it's just the idea about

端到端地控制这个东西，从硬件到软件，到模型环境，到模型权重，并且，你知道，收集所有这些数据来训练你的专用和更小的模型，这些模型随着你不断前进会变得更高效。是的，未来是伟大的，而且未来是本地化的。

controlling this thing end to end, from hardware to software to model environments to model weights, and, you know, collecting all of that data to train your specialized and smaller models that will be more efficient as, as you go. yeah, the future is great and the future is local.

是的，是的，绝对。你知道，确保这是你的数据、你的、你的权重、你的算力。我认为有一件非常有趣的事情是，我们最终在 DGX Spark 上确实获得了 10 倍的性能提升，我们给 Jensen、给高管团队、给我们正在帮助的团队发送了一份更新，在那封邮件中我非常喜欢的一句话是，我们没有为了解决这个问题而发明任何新的计算机科学。我们实际上采用了 Nvidia 的专家们已经解决并已经存在的东西，我认为我们一起做得非常好的事情是把它组装成一束花。我认为这说明了这里的一些可用性，对吧？我们在谈论什么是能力，什么，能力是什么，但你觉得仅仅是能力在阻碍人们采用这个吗？

Yeah, yeah, absolutely. you know, making sure it's your dates, your, your weights, your compute. I think one thing that was really interesting, we did end up getting 10x performance improvements on the DGX Spark, and we sent an update to Jensen, to the executive staff, to the teams that we're helping, and one line I really liked in that email was that we didn't solve any new computer science to do this. We actually took Things that the experts at Nvidia had already solved and was out there, and I think what we worked together to do really nicely was assemble it in a bouquet. And I think that speaks to some of the,the usability here, right? We're talking about what is capable, what, what are the capabilities, but Do you feel like it's just capabilities that are holding people back from adopting this?

是的。所以稍微讲一下这个故事，这里发生了什么。我想我们那周的周四吃了晚饭，然后，你知道，周五发了一封邮件，于是这个想法就出现了，比如，“我们为什么不做一个实验室呢，对吧？我们为什么不……”——你知道，这在当时对我来说听起来很疯狂。我没觉得这会成为可能，但就像是，你知道，我们——比如，Nada说：“哦，我们让一群Nvidia的人来帮你们，和你们一起工作，提升Spark上的性能。”对吧？然后我说，好吧，比如，好，我们——我是说，如果我们能做到，那就太好了。邮件发出去了，我想差不多就是那天晚上，然后周五Nada告诉我：“周一到这里来。”在Nvidia总部，你知道，我们会有人组成的团队在这里和你们一起合作做这件事。然后，你知道，我周一到了——Jensen谈到过“蜂群”这个概念。你知道，基本上就是整个公司会围绕某件事动员起来。

Yeah. So just to tell the story a little bit, what happened here, so I think we had dinner on Thursday of the week, and then, you know, an email got sent out on Friday, so this idea came up, like, "Why don't we do a lab, right? Why don't we, like, you know, this to me sounded crazy at the time. I didn't think this would be possible, but it was like, you know, let-- like, Nada was like, "Oh, let's like get a bunch of people from Nvidia to help you guys and work with you guys to improve the performance on the Spark." Right? And I was like, okay, like, let,yeah, let's-- I mean, if we can, that'd be great. Email got sent out, I think, pretty much that night, and then on Friday, Nada told me, "Be here on Monday." at Nvidia HQ, we're gonna have, you know, teams of people here that are gonna work together with you on this, and, you know, turned up on Monday and- Jensen talks about this concept of swarming. it's, you know, basically the idea that, the, the whole company will like mobilize around something.

你见过小孩子踢足球吗？有个球，所有人都扑上去。就像擒抱一样。这和足球完全相反。

Have you ever seen little kids play soccer? There's a ball, and everyone just attacks it. It's like a tackle. It's just the opposite of the soccer ball.

是的。我之前听说过这个想法，但你知道，我从未亲身体验过。我可以告诉你，我们那天真的是一个接一个，人们不断进出房间，你知道，来自各个中心、从事VLM工作的员工。我意识到，你知道，Nvidia每个方向都有一个团队，所以有一个“VLM for Spark”团队，这有点特别，但你知道，他们确实有所有这些团队。然后，你知道，我们基本上能够调动这些资源。而且，你知道，在大约三周内，如你所说，我们基本上获得了10倍的性能提升，对比Nvidia在Spark上现有的剧本中所运行的东西——那是使用Hermes agent的。所以我们在那里做了一系列优化，使用vLM作为推理后端。做了大量工作，比如调优模型、量化模型，让它们适合低调用。所以你会发现，我认为Nvidia在本地硬件上做得非常好的事情之一，就是它在数据中心运行的架构与在Spark上运行的架构相同，也就是Grace Blackwell，所以硬件基本上是相同的，这意味着你实际上可以免费得到很多东西。比如，内核已经非常好了。

Yeah. And I had heard about this idea, but like, I'd never experienced it. And I can tell you, like, we were just back to back that whole day, people coming in and out of the room, you know, from allCenter staff, people working, on VLM, I, I realized, you know, Nvidia has a team for everything, so there's like a VLM for Spark team, which is oddly specific, but like, you know, they have all these teams, and, you know, we basically got to pull those resources in and, you know You know, in those three weeks or so, as you mentioned, like we basically got 10x performance versus, you know, what, Nvidia had running on the Spark in their existing playbook, which was usingHermes agent, so we did a bunch of optimizations there, using vLM,as the, the sort of inference backend there. Doing a lot of work with the, like tuning the models, like quantizing the models, to, you know, be fit for low call. So what you'll find is, I think one of the great things NVIDIA has done Done with the local hardware is it's the same architecture that's running in the data center as running on the Spark, so it's Grace Blackwell, so the hardware is like fundamentally the same, meaning you actually get a lot of things for free. So for example, like the kernels are already really good.

不过，现在有很多调优和很多配置是专门为数据中心设计的。所以我们做的很多工作并不是发明什么新东西，而实际上只是调整东西，让它们在Spark上运行得更高效。而且，你知道，这个硬件能力极其强大。它是数据中心级别的硬件，却真的可以放在你桌上。所以关键在于，你知道，我们如何把它激活？我认为这是我过去几年学到的经验之一：我们已经拥有了硬件。硬件已经非常好了，而且模型也在变得更好。它们在压缩方面变得好多了，所以你可以把更多东西装进更小的设备里。所以，你知道，我认为这实际上是更多人来关注这个领域并投入其中的问题，然后，你知道，我们将能够做出惊人的事情。比如，我们有Nimitron three Ultra，这是一个五千五百亿参数的模型，在那边四台Spark上运行，每秒三十个token。所以这太巨大了。

However, there is like a lot of tuning and a lot of like configuration that is right now designed specifically for the data center. So a lot of the work that we did was not like inventing anything new, but it was actually just tweaking things To work, more performantly on the Spark, and, you know, the hardware is extremely capable. It's like, you know, it is data center level hardware that's literally can sit on your desk, so it's just about like, you know, how do we activate that? And I think this has been one ofmy learnings from the last few years is like, we already have the hardware. Like, the hardware is already really good, and, you know, the models are getting better. they're getting a lot better at compression, so you can fit more on a smaller device. So like, you know, I think it really is just a matter of more people looking at the space and working in it, and, you know, we will be able to do amazing things. Like, we have, you know, Nimitron three Ultra, it's a five hundred and fifty billion parameter model running on four sparks over there, thirty tokens per second. So like That's huge.

我想在这里说几句，就是，看到社区这么积极主动真的太好了。我想到了Alex Gmail，他就在这边，搞Exo，他是社区的一员，正在努力把开源智能的前沿推向新的高度。我很喜欢英伟达在这方面和他们合作。当我想到，你知道，这个领域的时候，我觉得，我们正处于Linux操作系统的九十年代，我们才刚刚开始。基础设施还不成熟。我们需要更多。我甚至还在说，我们可以合作。我们有一个开源的、系统部署系统，叫做ODS，它是一堆开源工具，我们部署到每台硬件上，配置东西，配置智能体，基本上，它能让你准备好，就像，你知道，能让你把本地智能体所需的整个端到端基础设施都搭好。这类东西，我们需要针对市面上每一种硬件进行优化，这样我们才能让越来越多的人加入进来，推动开源采用率，你知道，就是

I wanna say something here, like it's, it's so great to see the community being proactive. I think of Alex Gmail right here with Exo as somebody who's a member of the community who's trying to push the frontier of open source intelligence to the next level. I love how Nvidia is working with them on that. When I think about, you know, this space, I think, we're in the nineties of the Linux operating system, and we are like just starting. The infrastructure isn't there yet. We need so much more. I was even telling We can collaborate. We have like an opensource, sys-deployment system called ODS that is a bunch of open source tools that we deploy in each hardware, configure stuff, configure agents, and basically, gets you set, like, you know, get, gets you set up with the entire infrastructure end to end that you need for your agents locally. This kind of stuff, we need that Optimized for every piece of hardware out there so that we can get more and more people on board it, have the open source adoption, you know, just

一飞冲天。这就是我现在想要的。这就是我们在这里的原因。我们希望所有人都知道，本地、本地和开源AI可以运行在任何设备上，从你的手机到你的DGX Park，就像你刚才说的，Alex，到DGX Station，再到更高层次和数据中心，它会为你提供前沿级别的智能。

go to the moon. That's what I want right now. This is why we're here. We want everybody to know that local, local and open source AI can run on anything, starting from your phone to your DGX park, as you were saying, Alex, to DGX stations to the next level and the data centers, and it will deliver you frontier level intelligence.

是的，完全同意。我觉得，Matt，这个问题非常适合问你。你知道，你为大众受众测试所有东西。你听说了所有这些技术，而且你在大量使用。对普通用户来说，AI在哪里仍然存在不足？

Yeah, absolutely. I think This is a really good question for you, Matt. You know, you test everything for a mainstream audience. You're hearing about all of this technology, you're using this aggressively. Where does it still fall short for the, average user for AI?

所以我，我想到两件事：普通的个人用户，然后是企业用户使用开源。我的意思是，它，它，它，它基本上需要像打开Cursor一样简单。它需要比这稍微复杂一点，或者比安装CodeX并打开它稍微复杂一点。目前，公平地说，它离那个目标还很远。它，就像，这些家伙在做的事情令人难以置信。但它比大多数人，包括我自己，能够掌握的要复杂得多，你知道，更不用说企业了。我觉得，全职

So I, I think about two things, the average kind of personal user, then the average enterprise using, open source. I mean, it, it, it, it needs to basically be as simple as opening Cursor. It needs to be maybe slightly more complicated than that, or slightly more complicated than just installing CodeX and opening it up. Right now, it, to be fair, it is quite far from that. It, like, the, the stuff these guys are doing is incredible. But it is more sophisticated than what most people, includingmyself, are gonna be capable of, you know, let alone a, a business. I think full

时间来做这类事情都不过分。这就是为什么我们需要把它自动化。

time job to just do this kind of stuff. That's why we need to automate it.

而且它，它确实需要做到指向点击。一旦做到了那一步，而且，有很多很棒的开源项目，也有很多很棒的项目总体上正在接近那个目标，但我们还没有完全达到。让它被广泛采用的另一件事，是让人们能够——我们是否知道，它适合什么类型的模型、什么类型的测试框架、什么类型的硬件？确切地知道我能从我的家庭系统，或者从我在服务中心租的东西上获得什么用例，那也是非常重要的。而且

And it, it really does need to be point and click. And once it gets there, and there's, there's a lot of great open source projects, there's a lot of great projects in general that are getting there, but we're still not quite there. the other thing to make it widely adopted is to allow people toIs our appropriate for what, what type of model, for what type of harness, what type of hardware? Knowing exactly the use case that I can get out of my home system or something that I'm, I'm, you know, renting from service center, that, that is incredibly important as well. And

我，我觉得这不应该只写在文档里。我认为这就是开源对人们来说变得如此困难的地方。我认为它需要是指向点击，然后它自己就能搞清楚。是的。这就是ODS的意义所在。这就是我认为EXO的意义所在。你知道，随着我们不断发展壮大，构建这个基础设施，我们需要考虑的是普通日常用户的体验，而不是我们这些技术人员，因为我很确定我们都能自己搞定这些，但你的ChatGPT用户，你的云用户，不管是什么，他们想要的是一个替代方案，只需要点击、播放、发消息、用智能体，然后完事。

I, I think it shouldn't be just in documentation. I think that's where open source becomes so difficult for people. I think it needs to be a point and click, and it figures it out on its own. Yes. That's what ODS is about. That's what I think EXO is about. That's what, you know, as we grow more and more and, and building this infrastructure, we need to be thinking about the user experience for your average everyday user, not us as technical folks here, becauseI'm pretty sure that we all can manage our way around this stuff, but your ChatGPT users, your cloud users, whatever out there, they want this to be an alternative that is just click, play Send a message, use an agent, and done.

是的，大多数人，大多数人真的不想了解细节。他们，他们只想让它能用，即使它和Incurse或CodeX或任何其他工具无缝配合，然后它就能用了，他们不需要去考虑它。这是第一位的，对绝大多数人来说。

Yeah, most, most people really don't wanna know about the details. They, they just want it to work, and even if it works seamlessly with Incurse or CodeX or any of these other things, and it just worked, and they didn't have to think about it That is number one, that for, for the vast majority of people.

那么有趣的是，在一个多模型世界里，我们必须选择模型，你觉得那是产品层面的东西吗？还是说那是底层管道？那是需要关注的东西吗？你谈到了那个测试框架。

So interesting on that, in a multi-model world where we have to pick a model, do you think that that's something that is, product? as- Or, or is that, is that plumbing? Is that something that matter? You talked about the harness.

它，确实如此。它应该是某种，就像，无论某人AI体验的前端入口是什么，都应该是它在合适的时间为合适的用例选择哪个模型。这是一个非常困难的问题。有，你知道，开源项目、闭源项目、公司构建路由，那只是其中一部分——

It, it's exactly that. It should be some, like, whatever the kind of the front entry point to somebody's AI experience is, that should be what is choosing which model at the right time for the right use case. This is a very difficult problem. There are, you know, open source projects, closed source, companies building routing, that's only one-

其中的一部分，是的。很好，

Piece of it, yeah. Great,

所以继续说。不是像，Nvidia的Carter，他，他将是这里的主持人。嘿，Carter，看起来不错，老兄。所以，他，我，我，我跟他分享了所有步骤，他告诉我，“我喜欢它如何立即下载那个二十亿参数的模型，让我开始使用它，然后它开始下载下一个能在我的设备上完美运行的模型。”这就是我们需要给用户的那种体验，当我们……这些奇怪的东西，对普通用户来说工作量太大了。我们就那样失去他们。

so go ahead. Not like, Carter from Nvidia, he's, he's gonna be the moderator here. Hey, Carter, looking good, man. So, he, I, I, I shared all the steps with him, and he told me, "I love how, like, it immediately downloaded that two billion parameter model, allowed me to start playing with it, and then it started downloading the next model that would work perfectly on, on my device." That's the kind of experience we need to be giving these users when we These weird things,that is too much work for your average user. That's how we lose them.

是的，这很有趣，对吧？比如，当你不需要处理多模态，你只有一个普遍不错但——的大型模型，你几乎可以问它任何事。你不必那么老练，也不必太了解你的用例。但当我们谈论专业化模型时，如果你要专精于某件事，你需要知道那件事是什么。所以数量——我的意思是，你们能谈谈进入专业化模型或构建一个模型是什么感觉吗？或者你们有没有……

Yeah, it's interesting, right? Like the, when you're not having to deal with multimodal and you have just this generally good, but- A big model, you can kind of ask it anything. You don't have to be as sophisticated or you don't have to understand your use case too well. But as we talk about specialized models, if you're gonna specialize in something, you need to know what that something is. And so the amount that-- I mean Can you guys speak to maybe what it's like to move into a specialized model or to build one or have you, have

你——我可以先说说这个。在云端，你基本上得到的是正态分布。所以每次他们训练什么东西的时候，他们都在从所有人那里获取平均反馈，无论他们是高兴还是不高兴，或者，你知道，他们对答案满意，或者，“嘿，不，这不是我的意思，回去，或者换一个模型，用不同的模型回答。”那种反馈让你，你知道，云提供商能够微调下一个模型。但如果我们谈论的是针对用例的小型专业化模型，训练模型需要大量计算，存储这些权重也需要大量空间。如果没有每个用例或每个商业实体等专注于他们自己的模式、用例，以及他们如何处理智能体、他们的员工如何给这些智能体发消息、他们对什么工作流感兴趣，这不会发生。所以这真的是，你知道，每个用例不同，每个实体不同。这，这不是我们可以一概而论的事情，这就是为什么，你知道，持续学习并没有被前沿模型广泛谈论，但它即将到来，它会发生，

you-- I can start with that, on the cloud, you're basically getting the normal distribution. So every time that they are training something, they are taking the average feedback from everybody when they are being happy or sad or like, you know, they are okay with the answer or, hey, no, this isn't what I meant, go back or change the model and answer with a different model. That's the kind of feedback that allows you, you know, cloud providers to fine tune the next model. But if we're talking about small and specialized models for use cases, that's a lot of compute to train models, that's a lot of, storagefor these weights. that doesn't happen without each use case or each business entity, et cetera, focusing on their own, patterns and use cases and like how they handle agents and how their employees, message these agents and what workflows they're interested in. So it's really, you know, it's per use case, it's per entity. It, it's not something that we can just generalize, and that's why, you know, continual learning isn't being talked about by the frontier model, as much it's coming though, it will happen,

而且它要发生，就必须在本地硬件上运行。只有这样你才能拥有一个如此优化的东西，而不只是把Markdown写成一个文件放在一个仓库里，而且，你知道，你认为那个智能体不会忘记更新哪个规模、编辑什么或更新哪段记忆，因为在某个时候上下文窗口变得低效，这是一个问题。这就是当前智能体的范式，基本上只是保存到Markdown，下一个将是更新权重，而这必须在本地发生。

and it needs to be running on local hardware for it to happen. That's how you can have something that is so optimized that it's not just plotted markdown One file sitting in one repo, and, you know, you think that that agent isn't gonna lose track of which scale to update or what to edit or which memory to update, because at some point context lens becomes inefficient, and that's one issue. That's the current paradigm of agents is basically just saving to Markdowns, the next one will be updating the weights, and that needs to happen locally.

是的，这也有点好笑，因为，你知道，有一条推文，我想是Swix发的，内容是：“为什么微调即服务没有火起来？”大概是几个月前。我认为一个很大的原因是模型定制本身也是一个非常困难的问题。然后你能破解哪些模型？那是——我的意思是，这就是为什么Nvidia发布NeMoTron。它是一个开源模型，但从日期，抱歉，从数据到权重都开放。就是为了让它成为一个你知道可以安全使用和定制的模型。

Yeah, there's, it's funny too, 'cause, you know, there was a tweet, I think by Swix, that was like, "Why has fine-tuning as a service not taken off?" It was like a couple months ago. and I think a big reason for that is that model customization itself is also a very hard problem. And then what models can you hack on? that's-- I mean, that's why Nvidia releases NeMoTron. It's a open-source model, but everything from the dates, sorry, the data to the weights Just so that it is a model that you know you can safely,use and customize.

你知道，这很有趣，我一直在这个问题上摇摆不定。我，我想，比如，人们看通用模型有多好，你给它正确的上下文，那会比拥有一个微调模型以及随之而来的所有工作更好吗？我，我一直在来回纠结，因为能够拥有一个更小的、非常专业化的模型是非常有价值的，而且，也许一堆……但我，我想，是的，我不确定。你们，你们怎么看？我想，

You know, it's interesting, I keep kind of flip-flopping on this point. I, I think, like, people look at how good the generalized models are, and you give it the right context, is that gonna be better than, than having a fine-tuned model and all the work that comes with that? And I, I keep going back and forth 'cause there is a lot of value in being able to have kind of a smaller, very specialized model, and, and maybe a bunch ofbut I, I think, yeah, I'm not sure. What, what do you, what do you guys think about that? I think,

如果我能在这件事上插句话的话，我觉得，开源生态系统的美妙之处在于，所有这些路径都能得到探索，是的，然后无论哪个胜出，你知道，这就像推测解码发生的情况一样。你知道，有很多不同的方式来做推测解码，这个想法是你可以用一个较小的模型去近似一个较大的模型来加速它，而且，你知道，最近你看到了，老实说，我觉得就在上周，已经有三个不同的，相当，你知道，看起来像是推测解码方面的突破，它们来自不同的地方。一个是来自DeepSeek，还有一些工作是由MML和SG Lang团队完成的，他们为各种Qwen模型构建了D-Flash draft模型，这些模型相比以前的版本有了很大的改进。所有这些工作都在进行中。所以我们基本上能看到所有这些，而且，你知道，基本上无论最终是什么

if I can chime in on this one, like, I think the, the beauty of like the open source ecosystem is that like all of these paths kind of get to be explored, yeah, and then whatever wins, like You know, it's, it, it's like what happened with speculative decoding. you know, there's all these like different ways of doing speculative decoding, which is this idea that you can use a smaller model to basically approximate a larger model to speed it up and, you know, you've seen like recently, literally,I think in the last week, there's been like three different, like, quite, you know, seem to be breakthroughs in like speculative decoding that have just come from like different places. One from DeepSeek, there was some work that was done by like MML and, the SG Lang team to like build, D-Flash draft models for like various Qwen models that are, are like a big improvement on the previous ones. there's like all this work being done. And so like we kind, kind of get to see like all of it, and, you know, basically like whatever ends up being

最好的东西就会是那种胜出的东西，对吧？这是一个

the best thing will just be the thing that kind of wins, right? That's a

很好的观点，也许实际上并不是用户和消费者在定制他们自己的模型并使用这些专用模型，而是某个有需求的人这样做了，并且以开源的方式来做，这样其他人就可以直接采用它，对吧？如果我要开始为某个特定用例做图像生成，我只需要去找一个模型。实际上，我们也看到了很多这样的情况。我觉得

great point where maybe it's not users and consumers actually customizing their own models and using the specialized models, but someone who has a need does so and does so in an open source fashion so that someone else can just adopt it, right? If I'm gonna start to do image gen for a particular use case, I can just go find a model. Actually, we saw this a lot also As well. I think

最终终端用户关心的是，它能否解决我的用例，并且能否在我的预算范围内做到？是的。如果有一项服务能管理整个微调过程，而且我不需要去想——我知道我一直在说我不需要去想它，但有时候我确实会去想一些事情。但就像在这个特定的用例中，我在想

ultimately what the end user cares about is, does it solve my use case and can it do so within my budget? Yeah. And if there's another service that can manage the entire fine-tuning process and, and I don't have to think-- I know I keep saying I don't have to think about it, sometimes I do think about things. But like in this specific use case, I was thinking

已经太多事情了，不想再添一件。是啊，是啊。

about too many things already to add one more thing to it. Yeah, yeah.

就像，我，我关心我的业务，我不想考虑这个，

Like, I, I care about my business, I don't wanna think about the,

我可以描述一下我们常见的一种流程，就是蒸馏，或者通用模型、专用模型。所以如果你想一想，在很多情况下，就像——我喜欢提出的一个挑战，就像一个思想实验，就是，如果你告诉某人“想一个任何物体”，而“想一个你冰箱里的物体”，后者其实更容易，而且后者实际上是很多模型被部署到现实世界环境中的场景。所以，举个例子，你知道蒙特雷湾和蒙特雷湾水族馆研究所，他们叫MBARI。他们最近发现了一个新的鱼类物种，他们和Roboflow一起构建了流程来处理他们从深海潜艇上拍摄的所有水下视频。他们使用像Segment Anything 3这样的大模型和LLM作为裁判，基本上是说：“嘿，让我们拿所有这些视频，构建一个自动标注流程，尽可能多地理解我们收集的所有视频内容。”但最终，你知道，Sam知道所有关于从鱼到也许建筑图到，

I could, describe maybe a common flow that we see that's like distillation or, general model, specific model. So if you think about it, in a lot of cases, it's like if the The challenge I like to pose, that's like a thought experiment, is like, if you po-tell someone, "Think of any object," and it's like, "Think of an object in your fridge," the latter of those is actually easier, and the latter of those is actually where a lot of models kind of get deployed into real world settings. So, so for example, You know Monterey Bay and the Monterey Bay Aquarium Research Institute, MBARI, they're called. They discovered a new fish speciesrecently, and they built with Roboflow to process all of the underwater, Footage that they capture from their deep sea, submarines. And they use large models like Segment Anything 3 and LLMs as judge to basically say, "Hey, let's take all this video footage and let's build a auto-label pipeline to understand as many things as we can about all the video footage that we've collected." But then ultimately, you know, Sam knows everything about, you know, things from fish to maybe architectural diagrams to,

你冰箱里的物品以及介于两者之间的一切，但它们只关心那些，你知道，在这种情况下，是水下深海勘探的东西。所以我们看到的对于微调、最后一英里蒸馏来说非常常见的流程是：好吧，让我们采用一系列模型的大上下文，让这些模型也许都先通过一次理解某个东西，使用LM作为裁判来说：“嘿，我们达成了一定程度的共识，”现在我们有了一个专门的数据集，然后我们可以使用那个特定模型，实际上在那艘潜艇上实时运行它，并且还可以进行后处理以加快视频处理。而且如果你仔细想想，很多问题都是这种形态，就是说：“是的，我想要尽可能多的通用智能来理解这个东西，”然后最终我要解决的问题已经足够具体，即使不是完全无界的。而且我看到一个错误，相当——而且一个伟大的系列模型，就像，好吧，我应该采用SAM 3，然后也许只是微调SAM 3。在某种程度上，这实际上不太合理，因为你失去了让SAM 3出色的东西，也就是开放词汇能力。更好的做法可能是，如果你知道自己要蒸馏到一个特定的固定类别列表，那么你实际上可以去掉SAM那个庞大而昂贵的自动编码器部分，并使用一个特定的、也许像Deit或更专门的模型，取决于你要解决的任务，然后获得所有好处，你知道的，速度和准确性，同时仍然拥有准备的通用知识

items in your fridge and everything in between But they only care about things that are, you know, in this case, underwater deep sea exploration. And so a very common flow that we see for like the fine tune, last mile distillation is, okay, let's take the large context of an array of models Have those maybe all have a pass at understanding something, use LMs as judge to say, "Hey, we agree with some amount of consensus," and now we have a specialized dataset, then we can use that specific model and actually run that on the submarine in real time and also post-processing for faster video. And if you think about it, like a lot of problems areof that shape, where it's like, "Yeah, I want like general as much intelligence to understand the thing," and then ultimately the problem I'm solving is specific enough, even if not fully unbounded. And a mistake I see prettyAnd a great family model, it's like, okay, well, I should take SAM three and then maybe just fine tune SAM three. And in some ways, that actually doesn't make as much sense 'cause you lose the thing that makes SAM three awesome, which is the open vocabulary capabilities. What might be better is like if you know you're distilling down to a specific fixed class list, then you can actually drop the large, expensive auto encoder portion of SAM and use a specific, maybe like Deit or Or more specialized model depending on the task that you're solving and get all the benefits of, you know, speed up and accuracy, while still having the general knowledge of preparing

以及策划你的问题。你是，你是在为你的客户管理那个流程吗？呃，是的，工具让客户能够自己做。你是代表他们做呢？还是给他们工具让他们自己做？给他们平台让他们自己做。然后还有像配方一样的东西，是的，有人可以去照着做。然后像任何优秀的AI公司一样，有FDE，如果你需要，我可以卖给你，到后面来，他们可以为你做。

and curating your problem. And I think a lot of problems are of that shape Are you, are you managing that pipeline for your customer? The, yeah, the tooling makes it so they can do that. you do it on their behalf? or do you give them the tooling to do it? Give them the platform to do that. And then there's like recipes, yeah, where someone can go and do that. And then like any good AI company, there's an FDE that if you want, I can sell you, get around back here, and they can do it for you.

这，这，这基本上就是我之前谈到的用例和工作流程的一个很好的例子。当你试图找到，比如，你知道，为你的业务、你的企业边做边收集数据时，然后随着你的发展，决定如何在这些用例上微调一个模型，一个小的专门模型。这就是你如何变得更节省token的方法。

That, that, that's a great example basically of the use cases and workflows that I was talking about when you're trying to find, like, you know, collect the data as you go for your business, for your enterprise, and then decide how you're gonna fine-tune a model, a small specialized model on those use cases as you grow. That's how you become more token efficient

是的，是的。我，我，我认为今年和，和明年你会看到很多这样的用法：利用这些巨大的前沿模型来引导启动，你知道的，一个运行在开源上的更高效的设置。而且我认为，你知道，这很棒。就像，我认为这就是现在很多开源模型的构建方式，而且它，它，它已经被证明是，你知道，很难去阻止的。而且我只会鼓——你知道，鼓励人们也许，你知道，从这些前沿模型上转移开，但是，你知道，利用它们，利用它们来做什么呢？就是，就是用来引导启动，对吧？所以就像，

Yeah, yeah. I, I, I think this year and, and next year you're gonna see a lot of using these like monster frontier models to bootstrap, you know, like a more efficient setup that runs on open source. And I think, you know, that's great. Like, I think this is how a lot of the open source models have been built right now, and it's, it's, it's proving to be like quite hard to, you know, stop that. And I would just incu- you know, encourage people to just, you know, move away potentially from, like, you know, these, frontier models, but like, you know,use them, use them for what, for, for bootstrapping that, right? So like,

我喜欢你这么说。实际上，过去构建AI产品的方式是，我们会从机器学习开始，我们会先训练一个模型，然后我们会去寻找一个用例，而这些大模型让我们能够做的实际上是翻转过来。我们说我们可以先发现用例，然后如果合理的话再进入机器学习，这实际上，说实话，是Soex一个惊人的远见，因为他在几年前就为我们定义了这种模式，并且还举办了这次精彩的会议。

I love that you said that actuallyused to build AI products was we would start with the, with the machine learning, we would start by training a model, then we would go try to discover a use case, and what these big models allowed us to do is actually flip We said we could start with discovering a use case and then get into ML if it makes sense, and that was actually, honestly, an amazing foresight from Soex because he kind of defined that pattern, for us a few years ago and gave this amazing conference as well

是啊，这次会议在短短三年里发展得这么远，真是令人惊叹。

Yeah, it's crazy how far this conference has come in just three years.

嗯，完全同意。我有个问题。你们观众席里有问题吗？我们还有几分钟时间，我知道我们刚才在讨论开放提问，如果你们想直接问嘉宾，就直接喊出来。是的，请来吧。本地AI面临的重大开放性问题是什么？我觉得你们是这个领域的杰出人物，你们知道得更多，你们拥有最明亮的眼睛。

Yeah, totally. I have a question. Do you guys have questions in the, in the crowd? We have a few more minutes, and I know we were talking about Opening this up, if you guys wanna ask the panelists directly, just, just shout. Yeah. Please, go ahead. What are the big open problems in local AI? I feel like you're the luminaries of the field, you know more, you have the lightest eye

你想重复一下问题吗？我来重复那个问题。你刚才问的是：“本地AI的开放性问题是什么？” 我……

You want to repeat the question? I'll repeat that one. You just asked, "What are the open problems in local AI?" I,

我可以重复一遍。是的，问题是"本地AI最大的开放性问题是什么？"它仍然是优化问题，而在推理方面，仍然是如何让事情轻松启动，也就是你知道的，针对特定硬件的OSS、XOS。它仍然是我如何在预算限制和硬件限制下充分利用资源，以及如何从中榨取最大性能？你知道，那些模型——我仍然有同样的那几块三十系列显卡，以前用来跑Llama 2和Dolphin微调模型。现在它们在跑Quants 3.5、Quants 3.6，两百七十亿参数，性能出色，你知道的，更多内容在我的下一次演讲里。所以，你知道，归根结底是优化，而且我们仍然非常早期，还有空间容纳这么多玩家、这么多贡献者。我们需要所有能得到的帮助，来让这件事获得应有的成功，让我们把本地AI变成默认选项。这是我多年来的立场

I can repeat it. Yeah, it's, "What are the biggest open problems in local AI?" it continues to be optimizations and, for inference, it continues to be getting things easily kick-started, which is, you know, what OSS, XOS for specific hardware. It continues to be how can I make the most out of my budget constraints and hardware constraints and how can I squeeze the most performance out of that? you know, the, the models that, you know, I still have the same thirty ninetiesthat I used to run Llama two and like Dolphin fine tunes on. And now Now they're running Quants three point five, Quants three point six, twenty-seven billion parameters with excellent performance, you know, more on that in my next presentation. So, you know, it, it comes down to the optimization, and we are still very early that there is space for so many players, for so many contributors. We need all the help we can get to make this thing the success that it needs to be, that we need to make local AI the default. This has been my stance for years

我一直在说，开源AI必胜，大概从——一直以来都是如此，而我们做到这一点的方式，是给人们——无论是家里的个人、中型企业还是大型企业——一种轻松的方式，以非常高效的方式使用这些模型，不给他们带来比解决问题更多的麻烦。

now. I have been saying open-source AI must win for, like, since- Forever, and the way we do that is by giving the people, whether that's individuals at home, middle-sized businesses or enterprises, an easy way to use these models in a very efficient way that doesn't give them headaches more than solve their problems.

完全同意。如果你看看今天进行的这些圆桌讨论，那些就是我们相信最大的开放性问题，这也是为什么我们试图组建这个小组。所以，你知道的，有量化，都是关于讨论占用空间，这样模型才能装进这些小硬件里；有模型路由；还有模型本身，所以这些就是我们觉得现在的大问题，而随着我们举办未来的本地AI峰会，你知道，每次这些讨论话题都应该随着那些正在制约、或将要开启下一篇章的主题而改变。

Totally. And if you look at the panels that are, that are happening today, those are what we believe to be the biggest open-ended questions, which is why we tried to assemble this panel. And so the, you know, we have quantization, so all about talking about the footprint so that the models do fit on these, on these small hardware there's model routing, there's models generally, so those are kind of the, what we feel like are the big problems now, and as we do future local AI summits, I, you know, every time the panels should change to be the topics That are kind of holding the, or, or gonna be, gonnausher the next chapter in.

我可以补充一点吗？最大的挑战之一——我觉得有两个大挑战。一个是我们一直在讨论的，基本上就是简单性与可定制性之间的权衡。它是本地的，是你的，你可以做不同的事情，可以优化。如果是托管的，它开箱即用。那个权衡总是很困难。第二个，我实际上鼓励在座的各位去帮助解决的，是开放模型的重要性正日益受到质疑，而且我实际上认为，如果你觉得本地AI很重要，那么你就认为开源AI很重要，而真正重要的是，要成为能够使用、修改、适配和摆弄模型的倡导者。所以我认为，如果缺乏倡导，这个问题可能会越来越多地让我们感到失去控制。

Can I add one thing to that? One of the biggest challenges-- So I think there's two, two big challenges. One is what we've been talking about of basically the trade-off of simplicity versus customizability. It's local, it's yours, you can do different things, optimization. If it's hosted, it's built out of the box the way. That trade-off is, is always difficult. The second, which I actually encourage people in this room to help solve Is the importance of open models is becoming increasingly in question, and I actually think that like if you think local AI is important, then you think open source AI is important, and it's actuallyreally important to be an advocate for being able to use, change, adapt, and toy with models And so I think that that's a problem that, could increasingly be something that we feel less control over, absent advocating.

这是个很好的观点。我觉得这里的每个人都觉得——我对开源非常热情，这正是我们能够进入这个领域的全部原因。这就是这个领域取得进步的原因。这是一个必要的竞争环境。它让最好的想法能够传达给所有人。所以当有讨论说那是一种威胁的时候，你知道，我们需要投资并倡导开源，而且开源远比AI本身要大，对吧？就像，计算之所以在东海岸被发明，但硅谷却发生在这里，是有原因的，那是因为嬉皮士们意识到他们可以在软件中免费分享想法。我认为硅谷就是这种张力——想赚钱的资本家和想把想法免费送出去的嬉皮士之间的张力，而正是这种张力创造了如此不可思议的环境。

That's a great point. I think everyone here feels- I'm very passionate about open source, that is why we actually have access to this space at all. That's why the space has progressed. It's a, it's a necessary competitive environment. It allows for the best ideas to make their way to everybody so definitely when there's talks about that being a threat, you know, we need to invest and advocate for open source, and open source is so much bigger than AI, right? Like, there's a reason why computation, like, compute was invented in, on the East Coast, but Silicon Valley happened here, and it's because hippies realized thatthey could share ideas for free in software. I think Silicon Valley is this kind of tension between the capitalists that wanna make money and hippies that wanna give these ideas away for free, and that tension is what creates such an incredible environment here.

他们可以，而且

They canAnd

那就是这个领域运作得最好的时候。可以，

that's when the space works, works best. Can,

我可以就说，如果你关心这件事并且想更积极地参与，但也许，你知道，你不想涉足技术方面或者你不是技术人员，那么还是有办法参与的。所以有一个刚上线的网站叫 righttointelligence.org，这是一个让你参与进来、真正为开源发声、确保我们维护智能自由的方式。

can I just say like, if you care about this and you wanna be more of an active participant, but maybe, you know, you don't wanna get involved in the technical side or you're not technical, then there are ways to get involved. So there's a website that just came out called righttointelligence dot org, and this is a way for you to, like, get involved and actually, like, advocate for open source and to ensure that, you know, we maintain freedom of intelligence.

太棒了。我们的时间到了。为了以这种令人难以置信的方式开启本次活动，我们将带来精彩的演示。我们正在这个房间里运行基础智能模型。我们还将有几场精彩的、更多的圆桌讨论。这将是令人兴奋的一天，我们都还在这里逗留，所以请提问并积极参与。谢谢

Awesome. So we're out of time. For this incredible way to kick this off, we're gonna have amazing demos. We're running foundational intelligence models here inside of this room. we're gonna have a couple of amazing, a few more panels. It's gonna be an exciting day, and we're all lingering here, so ask questions and please participate. Thank you