【本地AI】手把手教你在电脑上跑AI模型 · 零基础指南
共 3 集
剧集目录
| 集 | 标题 | 时长 |
|---|---|---|
| P1 | 【本地AI】手把手教你在电脑上跑AI模型 · 零基础指南 中 | 24:26 |
| P2 | 【本地AI】手把手教你在电脑上跑AI模型 · 零基础指南 英 | 24:26 |
| P3 | 【本地AI】手把手教你在电脑上跑AI模型 · 零基础指南 无 | 24:27 |
P1 · 【本地AI】手把手教你在电脑上跑AI模型 · 零基础指南 p01 中英字幕 (P1)
字幕摘录
| 时间 | 英文 | 中文 |
|---|---|---|
| 0:00 | Everybody is talking about running AI locally, | 大家都在谈论本地的人工智能 |
| 0:03 | but almost nobody explains what it is | 但几乎没人解释是什么 |
| 0:05 | or how you actually get it to work. | 或者你如何真正让它工作。 |
| 0:07 | Now you get these videos where people throw around terms | 现在,你得到这些视频 人们扔周围的条件 |
| 0:09 | 如重量或量化、VRAM、GGUF、 | |
| 0:13 | and it sounds like you need a PhD | 听起来你需要博士学位 |
| 0:15 | and a $10,000 computer just to try this out. | 和一万块的电脑 只是试试这个。 |
| 0:18 | And that's exactly why most people give up | 这就是为什么大多数人放弃 |
| 0:20 | and just keep paying for subscriptions like ChatGPT. | 继续支付像ChatGPT这样的订阅费. |
| 0:23 | So let me give you the honest one sentence version here. | 所以,让我给你一个诚实的句子。 |
展开字幕全文（715 条）
| 序号 | 英文 | 中文 |
|---|---|---|
| 1 | Everybody is talking about running AI locally, | 大家都在谈论本地的人工智能 |
| 2 | but almost nobody explains what it is | 但几乎没人解释是什么 |
| 3 | or how you actually get it to work. | 或者你如何真正让它工作。 |
| 4 | Now you get these videos where people throw around terms | 现在,你得到这些视频 人们扔周围的条件 |
| 5 | 如重量或量化、VRAM、GGUF、 | |
| 6 | and it sounds like you need a PhD | 听起来你需要博士学位 |
| 7 | and a $10,000 computer just to try this out. | 和一万块的电脑 只是试试这个。 |
| 8 | And that's exactly why most people give up | 这就是为什么大多数人放弃 |
| 9 | and just keep paying for subscriptions like ChatGPT. | 继续支付像ChatGPT这样的订阅费. |
| 10 | So let me give you the honest one sentence version here. | 所以,让我给你一个诚实的句子。 |
| 11 | That is that local AI is simply a model file | 这就是说本地的AI只是一个模型文件 |
| 12 | that's sitting on your computer and a program that runs it. | 坐在你的电脑和运行程序上 |
| 13 | That's it. | 就这样 |
| 14 | No cloud, no API keys, no internet, no subscription. | 没有云,没有API键,没有互联网,没有订阅. |
| 15 | Everything else is really just a small detail. | 其他一切都只是个小细节 |
| 16 | So in this video, I'm gonna break down | 因此,在这个视频中,我要打破 |
| 17 | what's actually happening under the hood, | 引擎盖下到底发生了什么 |
| 18 | the real building blocks, and none of the fluff. | 真正的建筑构件, 和没有绒毛。 |
| 19 | And then I'm gonna show you how to run a model | 然后我会告诉你如何运行一个模型 |
| 20 | on your own machine in four completely different ways, | 在你自己的机器上 四种完全不同的方式, |
| 21 | from an app that you download | 从您下载的应用程序中 |
| 22 | all the way to pure low-level code | 一直到纯低层代码 |
| 23 | where you're running it yourself. | 在那里你运行它自己。 |
| 24 | Now by the end of this video, | 现在,在这个视频结束时, |
| 25 | you're gonna understand local AI | 你会明白本地的AI |
| 26 | better than most people that are posting about it. | 比大多数贴报的人要好 |
| 27 | So let's dive in. | 所以让我们潜入。 |
| 28 | So first, let's clear up | 先说清楚 |
| 29 | the most common confusion that I see. | 我所看到的最常见的混乱 |
| 30 | Now that's what's actually different | 这才是真正的不同 |
| 31 | between something like ChatGPT and a local model. | 在像ChatGPT之类的东西 和本地模型之间 |
| 32 | Now when you use ChatGPT, Cloud, Gemini, | 当你使用ChatGPT,云,双子座, |
| 33 | really any of these tools, | 任何这些工具, |
| 34 | what's happening is that you type a message, | 发生的事情是 你输入一个消息, |
| 35 | that message leaves your computer, | 留言留下你的电脑, |
| 36 | it travels over the internet to something like a data center | 它在互联网上游历到一个像数据中心的东西 |
| 37 | where a massive computer that you don't own | 你没有的大型电脑 |
| 38 | runs a giant model, | 运行一个巨大的模型, |
| 39 | and then the answer gets streamed back to your screen. | 然后答案被流回你的屏幕。 |
| 40 | So your computer did basically nothing. | 所以,你的电脑基本上什么都没做。 |
| 41 | It's really just a window into somebody else's machine. | 它真的只是一个窗口 进入别人的机器。 |
| 42 | Now local AI completely flips that. | 现在本地的AI完全推翻了这一点. |
| 43 | The model, so the actual file | 模型,所以实际文件 |
| 44 | that contains all of the intelligence | 包含所有情报 |
| 45 | gets downloaded onto your computer. | 下载到电脑上 |
| 46 | And when you ask it a question, | 当你问这个问题时, |
| 47 | your own CPU or GPU is the one that's doing the work. | 您的CPU或GPU是 做这项工作。 |
| 48 | So nothing leaves your machine, | 所以,没有什么离开你的机器, |
| 49 | and that gives you three big advantages. | 这给你三个大优势 |
| 50 | Now first, it's private. | 现在,首先,这是私人的。 |
| 51 | That's because your data never goes anywhere, | 那是因为你的数据从不消失 |
| 52 | it stays on your computer. | 它留在你的电脑上。 |
| 53 | Second, it's free because there's no subscription | 第二,免费,因为没有订户 |
| 54 | and there's no per-token cost. | 而且没有每桶成本。 |
| 55 | And lastly, it works offline. | 最后,它的工作离线。 |
| 56 | So if you're in a plane or a coffee shop with terrible Wi-Fi, | 所以,如果你在飞机或咖啡店 用可怕的无线, |
| 57 | doesn't matter, you can use these local models. | 无所谓,你可以使用这些本地模型。 |
| 58 | Now I do wanna be honest here about the trade-offs, | 现在,我想诚实地在这里 关于权衡, |
| 59 | which is that the models that you're gonna be able | 也就是你能够做的模型 |
| 60 | to run at home are much smaller than the frontier models | 在家跑步比边疆模式要小得多 |
| 61 | that you'd use from something like Cloud or OpenAI. | 你会用在云或OpenAI之类的东西上 |
| 62 | But they have gone shockingly good | 但是,他们已经走了惊人的好 |
| 63 | over the last couple of years, | 在过去几年里, |
| 64 | and for a huge amount of everyday tasks, | 和大量日常工作, |
| 65 | even coding tasks, they're more than enough. | 甚至编码任务,它们都足够了。 |
| 66 | So with that said, let's have a look | 所以说,让我们来看看 |
| 67 | at the actual pieces here, | 在实际的作品在这里, |
| 68 | so you understand local models much deeper. | 所以你更了解当地的模型 |
| 69 | Now the first part here is the model itself. | 现在第一部分是模型本身。 |
| 70 | And I wanna be really clear about what a model actually is, | 我想说清楚什么是模特儿 |
| 71 | because this is where some people imagine | 因为这就是一些人想象的 |
| 72 | some kind of magic black box. | 某种魔力黑盒。 |
| 73 | Now a model is literally just a file. | 现在一个模型其实只是一个文件。 |
| 74 | It's a really big file, could be hundreds of gigabytes, | 这是一个非常大的文件,可能是数百千兆字节, |
| 75 | that's full of numbers that we call weights. | 我们称之为重量数 |
| 76 | Now these are billions of numbers | 这些是数十亿的数字 |
| 77 | that got baked in when the model was trained. | 当模特被训练时 被烤熟了 |
| 78 | Now that file doesn't think, it doesn't run, | 现在那个文件不思考,它不运行, |
| 79 | it just sits there on your disk | 它只是坐在你的磁盘上 |
| 80 | like any other file that you would have. | 就像其他文件一样 |
| 81 | And companies like Meta, Google, Alibaba, Mistral, whatever, | 还有梅塔、谷歌、阿里巴巴、米斯特拉尔等公司 |
| 82 | they release these files for free. | 他们免费发布这些文件。 |
| 83 | Now those are your open models | 这些是你的公开模型 |
| 84 | and things that you keep hearing about | 和你们所听到的 |
| 85 | 比如Llama或者Gemma或者Quen或者DeepSeq,你懂这个意思. | |
| 86 | You can literally just download them | 你可以直接下载 |
| 87 | because they are literally just a file | 因为从字面上来说,它们只是一个文件 |
| 88 | full of billions of different numbers. | 充满了数十亿个不同的数字。 |
| 89 | Now when you go to look at these models, | 当你去看这些模型时, |
| 90 | you're gonna see names like four billion, | 你会看到40亿的名字, |
| 91 | eight billion, 70 billion. | 八十亿七百亿. |
| 92 | Now that B is billions, | 现在B是亿, |
| 93 | and it stands for the number of parameters. | 它代表参数的数量。 |
| 94 | Now that's effectively just how many numbers | 现在,这是有效的 只是多少数字 |
| 95 | are inside of that file. | 里面的档案。 |
| 96 | And the rule here is pretty simple. | 这里的规则很简单。 |
| 97 | Generally speaking, more parameters means a smarter model, | 一般来说,更多的参数意味着一个更聪明的模型, |
| 98 | but it also means a bigger file | 但它也意味着一个更大的文件 |
| 99 | that needs more memory and compute to run. | 这需要更多的记忆和计算运行。 |
| 100 | So if you look at an eight billion model, | 所以如果你看看80亿的模型 |
| 101 | this is maybe a few gigabytes in storage. | 这也许是几千兆字节的存储。 |
| 102 | Well, a 70 billion model is a file that's so large | 嗯,一个700亿的模型是一个文件 这是那么大 |
| 103 | that most laptops simply cannot even load it, | 大部分笔记本电脑都装不下 |
| 104 | and some may not even have enough storage to download it. | 而有些人甚至可能没有足够的存储器可以下载. |
| 105 | Now that leads directly into the biggest trick | 现在,这直接 进入最大的诡计 |
| 106 | that happens with local AI. | 这发生在本地AI。 |
| 107 | Now this is something called quantization. | 这是所谓的量化。 |
| 108 | Now it sounds scary and really complicated, | 现在听起来很可怕,很复杂 |
| 109 | but it's really the same idea as compressing a photo. | 但它真的相同的想法 压缩一张照片。 |
| 110 | So you can take these billions of numbers | 这样你就可以拿这些亿万的数字 |
| 111 | and you can store them with less precision. | 并且可以更精确地储存它们。 |
| 112 | And that means that this model file | 这意味着这个模型文件 |
| 113 | is gonna get dramatically smaller | 就会变小 |
| 114 | with barely any quality loss at all. | 几乎没有质量损失。 |
| 115 | So a model that would normally need 16 gigabytes of memory | 所以这个模型通常需要16千兆字节的内存 |
| 116 | in its original form might only need five, six, | 原型可能只需要五,六, |
| 117 | or seven gigabytes after quantization. | 或七千兆字节后 |
| 118 | And when you see the term gguf floating around, | 当你看到这个术语漂浮, |
| 119 | that's basically just the standard file format | 这基本上只是标准文件格式 |
| 120 | for these compressed models. | 这些压缩模型。 |
| 121 | Now this is the entire reason that normal computers | 这才是正常电脑的原因 |
| 122 | can even run AI models at all, | 甚至可以运行AI模型, |
| 123 | or at least some of the bigger ones. | 或至少一些较大的。 |
| 124 | So when you hear about quantization | 所以,当你听到关于量化 |
| 125 | or a quantized model, | 或被量化的模型, |
| 126 | just think of that as a compressed model | 把这当成一个压缩模型 |
| 127 | that's meant to make it smaller so it's easier to run. | 意思是让它变小 所以它更容易运行。 |
| 128 | When you compress these models, | 当你压缩这些模型时, |
| 129 | you keep almost the exact same performance, | 你保持几乎完全相同的表现, |
| 130 | but again, you just reduce the size drastically, | 再说一次,你只是大大缩小了尺寸 |
| 131 | so it's a lot easier to actually run. | 因此,这很容易真正运行。 |
| 132 | Now the next piece is the inference engine. | 现在下一个是推论引擎。 |
| 133 | And this is the part that almost nobody explains. | 这是几乎没人解释的部分。 |
| 134 | Remember that the model is just a file full of numbers, | 记住,模型只是一个文件 充满数字, |
| 135 | and a file can't run itself. | 并且文件不能自己运行。 |
| 136 | So you need a program | 所以你需要一个程序 |
| 137 | that can actually load these numbers into memory | 可以将这些数字装入内存 |
| 138 | and do the actual math to make the model work | 并做实际的数学 使模型工作 |
| 139 | and predict the next token. | 并预言下一个迹象。 |
| 140 | Now that program is called an inference engine. | 现在这个程序叫做推论引擎. |
| 141 | Now the most famous one is called llama.cpp. | 现在最著名的一个叫做lama.cpp. |
| 142 | And here's the secret, | 这是秘密 |
| 143 | almost every tool that I'm gonna show you today, | 几乎每一个工具 我今天要给你看, |
| 144 | so LM Studio, Olamma, Docker Model Runner, | 因此LM工作室,奥拉姆马,多克模型运行器, |
| 145 | they're all basically just a wrapper | 他们基本上都只是一个包装器 |
| 146 | around engines like this. | 绕着这种引擎 |
| 147 | Now the engine does all of the work, | 现在引擎完成了所有的工作, |
| 148 | and the tool just kind of makes it nice to use, | 而工具只是 那种使它使用, |
| 149 | which you're gonna see later on. | 稍后你会看到的。 |
| 150 | Now the last building block here is your hardware, | 最后一块是你们的硬件 |
| 151 | because there's really only one question | 因为只有一个问题 |
| 152 | that decides what local models | 决定何种地方模式 |
| 153 | you're going to be able to run. | 你会跑的 |
| 154 | Now that is, how much memory do you have on your computer, | 你电脑里有多少记忆 |
| 155 | and how fast is that memory? | 记忆有多快? |
| 156 | Now on a PC with a graphics card, | 现在在一台带显卡的PC上 |
| 157 | so a dedicated graphics card, | 所以一个专用的显卡, |
| 158 | that number is gonna be your VRAM. | 这个号码会是你的VRAM。 |
| 159 | So if you have an Nvidia 3090 or 4090 or 5090, | 所以,如果你有一个Nvidia 3090或4090或5090, |
| 160 | or I'm just naming random GPUs, | 或者我只是命名随机GPU, |
| 161 | you're gonna look at the VRAM on that device. | 你看看那个装置上的VRAM |
| 162 | Now if you're running on a modern Mac computer, | 如果你在现代Mac电脑上运行 |
| 163 | 就像M3或M4一样 任何M系列的MacBook | |
| 164 | then you're just gonna be looking at the amount of RAM | 那你就只能看看内存的数量 |
| 165 | that your computer has. | 你的电脑有。 |
| 166 | And that's because Apple shares its memory with the GPU, | 这是因为苹果公司与GPU共享其记忆, |
| 167 | and has something called unified memory. | 并有所谓的统一记忆。 |
| 168 | Now other devices have different specifications, | 现在其他设备有不同的规格, |
| 169 | but generally speaking, | 但总的来说, |
| 170 | if you have a relatively new computer, | 如果你有一个相对新的计算机, |
| 171 | if it has a dedicated graphics card, | 如果它有一个专用的图形卡, |
| 172 | 你看着VRAM。 | |
| 173 | That's the amount of memory you're gonna have | 这就是你的记忆量 |
| 174 | for running local models. | 用于运行本地模型。 |
| 175 | And if you're on a Mac, again, a modern one, | 如果你在Mac上 又是现代的 |
| 176 | you're looking at the amount of unified memory. | 你在看统一记忆的量 |
| 177 | And the rule of thumb here is pretty simple. | 这里的拇指规则很简单。 |
| 178 | The model file needs to be able to fit | 模型文件需要适合 |
| 179 | inside of that memory that you have | 在你的记忆中 |
| 180 | with a little bit of room to spare. | 还有一点空位 |
| 181 | So roughly speaking, | 所以说, |
| 182 | if you have eight gigabytes of VRAM or unified memory, | 如果你有8千兆字节 VRAM 或统一的内存, |
| 183 | you're gonna be able to run three to four billion | 你会跑到三四十亿 |
| 184 | parameter models, even without being quantized. | 参数模型,即使没有被量化. |
| 185 | And if you have 16 gigabytes of memory, | 如果你有16千兆字节的记忆 |
| 186 | you can go up to seven or eight billion parameter models. | 可以达到70或80亿个参数模型。 |
| 187 | And then if you go up to 32 gigabytes, | 然后如果你升到32千兆字节 |
| 188 | you start to be able to get into the 14 | 你开始可以进入14 |
| 189 | to 30 billion parameter range. | 到300亿参数范围。 |
| 190 | And this is where things start to feel genuinely smart, | 而这就是事情开始 感觉真正聪明, |
| 191 | especially for local models. | 特别是当地模特儿 |
| 192 | But keep in mind, | 但记住, |
| 193 | you don't need a monster computer to do any of this. | 你不需要怪物电脑来做这些 |
| 194 | Even with something like your phone, | 即使是用你的手机一样的东西 |
| 195 | you can run small models already. | 你可以运行小模型了。 |
| 196 | And one thing that carries over from cloud AI | 还有从云中传来的一件事 |
| 197 | is that the context windows | 是上下文窗口 |
| 198 | to the model short-term working memory | 到模式短期工作记忆 |
| 199 | is going to actually affect the amount of space | 将会影响空间的大小 |
| 200 | that's being taken up. | 这正在处理中。 |
| 201 | So if you have long conversations, bigger documents, et cetera, | 所以,如果你有长话短说, 更大的文件等等, |
| 202 | that's also going to fill up your memory. | 那也会让你的记忆充满 |
| 203 | Now I'm gonna explain this a little bit more in detail | 现在我要详细解释一下 |
| 204 | because this is super important. | 因为这很重要 |
| 205 | But the basic idea is that the one number | 但基本想法是,一个数字 |
| 206 | you need to be aware of is how much memory, | 你需要知道 有多少记忆, |
| 207 | either VRAM or unified memory is on your computer. | 无论是VRAM还是统一的内存都在你的电脑上。 |
| 208 | When you look at a model, whether it's quantized or not, | 当你看一个模型, 无论它是否被量化, |
| 209 | it needs to be able to fit comfortably | 它需要能够舒适地装配 |
| 210 | within that memory range. | 在那个记忆范围内 |
| 211 | So if you have 32 gigabytes of VRAM | 所以,如果你有32千兆字节 VRAM |
| 212 | and a model is 25 gigs, that's fine. | 一个模型是25个工作,没关系。 |
| 213 | You'll be able to run it, right? | 你会运行它,对不对? |
| 214 | However, one thing to keep in mind with memory | 不过,有一件事要记住 |
| 215 | is also the speed of the memory. | 也是内存的速度。 |
| 216 | So while you will be able to run much larger models | 所以当你可以运行 更大的模型 |
| 217 | on things like modern Macs that have 128 gigabytes | 在像现代Macs这样的 有128千兆字节的东西上 |
| 218 | of unified memory, the inference speed of those models | 这些模型的推论速度 |
| 219 | is gonna be much slower than a similar model | 比一个类似的模型慢得多 |
| 220 | that's running on something like a dedicated GPU | 运行在像一个专用GPU的东西上 |
| 221 | 或Nvidia RTX GPU. (中文(简体) ). | |
| 222 | The reason for this is the speed of the memory. | 其原因是记忆的速度. |
| 223 | So while the memory will dictate the size of the model | 所以虽然记忆会决定模型的大小 |
| 224 | and the more memory you have, | 越多的记忆, |
| 225 | the smarter models you can run, | 你能够运行的更聪明的模型, |
| 226 | the memory speed matters for the tokens per second | 每秒符号的内存速度很重要 |
| 227 | and the inference speed that you're going to be able | 和推论速度 你将有能力 |
| 228 | to generate. | 生成。 |
| 229 | There's a lot of different techniques here | 这里有很多不同的技术 |
| 230 | and things that I could get into, | 和我可以进入的东西, |
| 231 | but generally speaking, | 但总的来说, |
| 232 | if we talk about dedicated graphics cards, | 如果我们谈论专用显卡, |
| 233 | again, typically in the Nvidia family, | 通常在恩维迪亚家族, |
| 234 | these are much faster, sometimes two or three times faster | 速度快得多 有时快两三倍 |
| 235 | at inference speed, but they usually have less capacity. | 在推论速度上, 但他们通常能力较低。 |
| 236 | So for example, I have 24 gigabytes of VRAM in my 4090 | 比如,我4090年有24千兆字节的VRAM |
| 237 | and it's very fast and can generate 200 tokens per second | 它的速度很快,每秒能产生200个令牌 |
| 238 | for some of the models that I run. | 我运行的一些模型。 |
| 239 | However, I can't run models that are 70 billion parameters | 但我不能运行700亿参数的模型 |
| 240 | like I might be able to on my Mac. | 碞钩ии沉 |
| 241 | However, on my Mac, those models are really slow | 然而,在我的Mac上,这些模型真的很慢 |
| 242 | because the memory speed is significantly slower. | 因为内存速度明显较慢. |
| 243 | So you're going to be looking at memory speed | 所以你要看记忆速度 |
| 244 | as well as memory capacity. | 以及记忆能力。 |
| 245 | Those are the two things that are going to dictate | 这些是两件事要决定的 |
| 246 | what you can do with local AI. | 你对本地的AI能做什么? |
| 247 | And there's always going to be a trade-off | 总是有取舍 |
| 248 | in terms of the size of the model | 从模型的大小来看 |
| 249 | and the inference speeds that you're getting. | 和你的推论速度。 |
| 250 | Typically smaller models are going to be much faster. | 通常较小的模型会更快 |
| 251 | And again, the most important thing | 再说一遍,最重要的是 |
| 252 | is that whatever model you run, | 是无论你运行的模型, |
| 253 | it needs to fit in this memory | 它需要融入这个记忆 |
| 254 | while it still can run if it's not in there, | 如果里面没有它 还能运行 |
| 255 | it's going to be so slow that it's practically unusable. | 速度会太慢 以至于无法使用 |
| 256 | Generally speaking, if you're looking at models | 一般来说,如果你在看模特儿 |
| 257 | between 14 and 35 billion parameters, | 140亿至350亿参数之间, |
| 258 | those are going to be a really good sweet spot. | 这将是一个非常好的甜点。 |
| 259 | They're going to give you pretty decent performance | 他们会给你一个很好的表演 |
| 260 | and you're not going to feel | 你不会觉得 |
| 261 | like you're missing out on too much. | 就像你错过了太多。 |
| 262 | If you go up to huge models, | 如果你去大模型, |
| 263 | like 120 billion parameters, 250 billion parameters, | 比如1200亿参数,2500亿参数, |
| 264 | you're talking about needing extremely high-end hardware | 你说的是需要非常高端的硬件 |
| 265 | running at slow inference speeds. | 以缓慢推论速度运行。 |
| 266 | And it's very difficult to actually run those | 很难真正运行这些 |
| 267 | at scale on your own machine. | 在自己的机器上。 |
| 268 | Anyways, with that in mind, let's keep going here. | 无论如何,考虑到这一点, 让我们继续在这里。 |
| 269 | I want to tell you about something really interesting. | 我想告诉你一些有趣的事 |
| 270 | Now the whole reason you'd even bother | 现在的整个原因 你甚至会烦恼 |
| 271 | running models locally comes down to one main thing, | 本地运行的模型 归根结底是一件事, |
| 272 | which I know you all want, which is control. | 我知道你们都想要的 这就是控制 |
| 273 | You pick the model, you own the setup | 你选模特儿,你自有一套 |
| 274 | and nobody can change the deal in. | 没有人可以改变交易。 |
| 275 | But here's the problem, right? | 但问题是,对吧? |
| 276 | Almost every tool that you'd actually want to use | 几乎每个工具 你真正想要使用 |
| 277 | locks you into one provider's model. | 把你锁在一个供应商的模型里 |
| 278 | Now that's exactly what Mindhub Cowork, | 这正是明德胡布同事 |
| 279 | who's the sponsor of today's video, is built to fix. | 谁是今天录影带的赞助商 是用来修复的 |
| 280 | Now it's open source, free to use, | 现在它是开源,自由使用, |
| 281 | and it has a real model router. | 它有一个真正的模型路由器。 |
| 282 | 你可以经营Claw、GPT、Chaminade、Jeepseq | |
| 283 | or the same local models that I've been talking about | 或者我一直在谈论的本地模特儿 |
| 284 | right here, all inside of this workspace. | 在这里,所有在这个工作空间内部。 |
| 285 | So the workflow is simple. | 所以工作流程很简单。 |
| 286 | You brief the built-in agent harness, which is Anton, | 你向内置特工介绍一下 也就是安东 |
| 287 | walk away and come back to finished work. | 走开,回来完成工作。 |
| 288 | Now I asked it to research the latest coding models | 现在我要求它研究最新的编码模型 |
| 289 | and build me a comparison dashboard. | 帮我造一个比较仪表板 |
| 290 | And this is what it came back with, | 这就是它回来的原因 |
| 291 | an actual dashboard that I can open and share, | 一个真正的仪表板 我可以打开和分享, |
| 292 | not just a wall of text. | 不只是文字墙。 |
| 293 | And this is the part that connects | 这是连接的部分 |
| 294 | to everything in this video. | 给这段录像里的所有内容 |
| 295 | When a better model drops, | 当一个更好的模型下降, |
| 296 | whether it's local or a cloud model, | 无论是本地还是云型 |
| 297 | I can swap it in and I don't have to change anything. | 我可以把它换进去,我不需要改变任何东西。 |
| 298 | So I have the same workspace, the same work, | 所以我有相同的工作空间,相同的工作, |
| 299 | and since it's fully open source, | 既然它完全开源 |
| 300 | you can clone the repo, | 你可以复制Repo, |
| 301 | spin it up locally in just a few commands, | 转动到局部 仅几个命令, |
| 302 | or just download the dedicated Mac or Windows app. | 或者只下载专用的Mac或Windows应用程序. |
| 303 | The whole thing runs on your own hardware, | 整个事情运行在你的硬件上, |
| 304 | which is basically the end game | 这基本上是结束游戏 |
| 305 | of what we're testing today. | 今天的测试 |
| 306 | Now they also have a hosted version, | 现在,他们还有一个主机版本, |
| 307 | but honestly, I love the desktop app | 但老实说,我喜欢桌面应用 |
| 308 | as it's very easy to use. | 因为它很容易使用。 |
| 309 | So I'm gonna leave a link to it in the description. | 所以我要在描述中留下一个链接。 |
| 310 | Try it out. | 试试看 |
| 311 | Again, it really goes nicely with these local models. | 再一次,它真的 与这些当地模型。 |
| 312 | And now let me show you | 现在让我告诉你 |
| 313 | how we can actually run local models. | 如何实际运行本地模型。 |
| 314 | So there's a lot of ways to run a local model. | 因此当地模式的运行方式很多. |
| 315 | And just like anything in software, | 就像软件里的任何东西一样 |
| 316 | it really comes down to how much control you want. | 这真的取决于你想控制多少 |
| 317 | So I've broken this into four different tiers. | 所以我把它分为四层。 |
| 318 | Now at the top, we've got LM Studio. | 现在最上面的是LM工作室 |
| 319 | This is a regular desktop app. | 这是普通的桌面应用程序 。 |
| 320 | You can click, you can download, you can touch everything, | 你可以点击,你可以下载, 你可以触摸一切, |
| 321 | and you don't really need to go into the terminal. | 而你真的不需要进入终端。 |
| 322 | Now below that, we have Olamo. | 下面是奥拉莫 |
| 323 | This is a really popular option, especially for developers, | 这是一个非常流行的选择, 特别是对开发者来说, |
| 324 | and it's just one command inside of your terminal | 它只是一个命令 在你的终端内部 |
| 325 | where you can talk with models, spin up a local server. | 您可以与模型交谈, 旋转本地服务器。 |
| 326 | You get the idea. | 你得到这个想法。 |
| 327 | 然后我们有了Docker Model Runner。 | |
| 328 | Now this is really good | 现在,这是真的好 |
| 329 | because it treats models like containers, | 因为它对待模型 像容器, |
| 330 | which is perfect if you're actually gonna be deploying | 如果你真的要部署,那就完美了 |
| 331 | these alongside live applications. | 它们与现场应用同时存在。 |
| 332 | And at the bottom of my list here, we have full code. | 在我的名单底部, 我们有完整的代码。 |
| 333 | This is where you're running a model in pure Python | 这就是你运行一个模型 在纯Python |
| 334 | and you see every single piece. | 你可以看到每一块。 |
| 335 | Now, no matter which way you wanna run these models here, | 现在,不管你想用哪种方式运行这些模型, |
| 336 | you're gonna be making three decisions. | 你会做出三个决定 |
| 337 | You're gonna pick a model, | 你要选一个模特儿 |
| 338 | you're gonna pick a size and quantization | 你会选择一个大小和量化 |
| 339 | that fits inside of your memory, | 这符合你的记忆, |
| 340 | and you're gonna decide how you wanna talk to it. | 你要决定怎么跟它说话 |
| 341 | So whether that's a chat window or something like code. | 所以,无论这是一个聊天窗口 还是类似代码的东西。 |
| 342 | If you keep that in mind, | 如果你记住这一点, |
| 343 | every single one of these tools is gonna make sense. | 这些工具中的每一个都会有意义 |
| 344 | So let's start at the top. | 故从上起. |
| 345 | And again, I'm gonna go through all of them | 再说一遍,我要翻遍他们所有 |
| 346 | and show you exactly how to run local models. | 告诉你如何运行本地的模型。 |
| 347 | Let's dive in. | 让我们潜入。 |
| 348 | So the first tool on my list here is LM Studio. | 我的第一个工具是LM工作室。 |
| 349 | I'll give you a quick walkthrough, | 我给你一个快速的走过, |
| 350 | but this is one of the best ways to run local models. | 但这是管理本地模型的最佳方法之一 |
| 351 | Now, once you download the tool again, it's completely free, | 现在,一旦你再次下载工具, 它是完全免费的, |
| 352 | you're gonna be brought into a view | 你会被带入风景 |
| 353 | that looks something like this. | 看起来像这样 |
| 354 | From here, you're gonna go into the model view, | 从这里开始,你会进入模型视图, |
| 355 | looks like kind of a robot icon on the left-hand side, | 看起来像是左边的机器人图标 |
| 356 | and you'll be able to search | 你就可以去寻找 |
| 357 | through all of the available models | 通过所有可用的模式 |
| 358 | that you can download directly here. | 您可以在此直接下载。 |
| 359 | Now, what you'll wanna do is search for a model | 现在,你想做的是寻找一个模型 |
| 360 | that matches the kind of relative size | 匹配相对大小的类型 |
| 361 | or amount of memory that you have. | 或你的记忆量 |
| 362 | Again, if we're talking about larger high-end machines, | 再说一遍 如果我们说的是更大的高端机器 |
| 363 | you can typically get away with 27 billion, 35, | 你通常能逃出270亿 35亿 |
| 364 | 30 billion parameters, et cetera. | 三百亿参数等. |
| 365 | If you have eight gigs of RAM or 16 gigs of RAM, | 如果你有8个工作 或16个工作 RAM, |
| 366 | look for ones that are eight billion parameters | 寻找80亿的参数 |
| 367 | or four billion parameters, much smaller sizes. | 或40亿参数,大小小得多。 |
| 368 | So for example, we have Gwen 3.8. | 例如,我们有格温3.8。 |
| 369 | You'll notice that if I click on this, | 你会注意到,如果我点击这个, |
| 370 | I can view different levels of quantization | 我可以看到不同层次的量化 |
| 371 | and see the change in size right here. | 看这里大小的变化 |
| 372 | You also see kind of some icons or indications | 您还看到一些图标或指示 |
| 373 | of which model is the best for you based on your hardware. | 根据你的硬件,哪个型号最适合你。 |
| 374 | So you can see when it says full GPU offload possible | 所以你可以看到,当它说全GPU卸载的可能性 |
| 375 | and a little thumbs up, | 并竖起一小拇指, |
| 376 | that's the one that you would wanna go with. | 那是你想去的那个 |
| 377 | And notice this number of Q4, | 注意这个Q4的号码 |
| 378 | that's the level of precision or the quantization level. | 这是精确度或量化度。 |
| 379 | So the lower, so like Q4, Q2, Q1, | 所以越低,就像Q4,Q2,Q1, |
| 380 | the more quantization you have, right? | 越多的量化,对不对? |
| 381 | So if you have Q8, you can see this is bigger | 所以,如果你有Q8,你可以看到 这是更大的 |
| 382 | than Q6 or Q4, and it's a pretty drastic difference. | 比Q6或Q4, 这是一个相当巨大的差异。 |
| 383 | You're also gonna wanna look at the capabilities, | 你也想看看他们的能力 |
| 384 | so vision, tool use, reasoning. | 因此,视觉,工具的使用,推理。 |
| 385 | If you need it to be able to analyze images, you need vision. | 如果需要它才能分析图像,就需要视觉. |
| 386 | If you want it to work in an agentic harness | 如果你想让它在 代理化的绳子上工作 |
| 387 | or an agent mode, you need tool calling. | 或代理模式,您需要工具呼叫。 |
| 388 | Hopefully you get the idea. | 希望你知道这个主意 |
| 389 | There's so many models, | 这么多模特儿 |
| 390 | I can't possibly go through all of them, | 我不可能通过他们所有的, |
| 391 | but you get the idea you can browse through here | 但你有这个想法 你可以浏览这里 |
| 392 | and look at the ones that are going to match | 看看那些会匹配的 |
| 393 | your specific hardware and the sizes that make sense. | 您的特定硬件和大小 合理。 |
| 394 | Now, once you download the model, | 现在,一旦你下载模型, |
| 395 | you can view your models from the model tab right here. | 您可以在这里查看模型标签。 |
| 396 | Now, in order to use these models, | 现在,为了使用这些模型, |
| 397 | you do need to load them. | 你确实需要加载它们。 |
| 398 | So if I select a model like Gemma 4, | 所以如果我选一个像Gemma4的模特儿 |
| 399 | and I just bring open this sidebar here, | 我只是把这个侧柱打开, |
| 400 | it will give me some options for actually loading | 它会给我一些选项 实际加载 |
| 401 | and running my model. | 和运行我的模型。 |
| 402 | So what I may wanna do here is go to the load tab | 所以,我想在这里做的是 去负载标签 |
| 403 | and start changing some of the values. | 并开始改变一些值。 |
| 404 | If you're a beginner, don't change anything | 如果你是初学者,不要改变任何东西 |
| 405 | and just run this directly. | 然后直接运行这个。 |
| 406 | The one thing you can't have a look at | 只有一件事你无法看 |
| 407 | is the context size here. | 这里为上下文大小。 |
| 408 | And keep in mind, the larger you make this context size, | 记住,你做的这个背景尺寸越大, |
| 409 | the more room is gonna be taken up in your computer's memory | 在你的电脑记忆中,会占用更多的房间 |
| 410 | because all of this context needs to actually fit | 因为所有这些背景都需要实际适应 |
| 411 | in the GPU memory, right? | 在GPU的记忆,对不对? |
| 412 | Or the memory that you have for running local models. | 或者你运行本地模型的记忆。 |
| 413 | There's a bunch of other settings you can use, | 还有一堆其他的设置你可以使用, |
| 414 | but in this case, | 但在这种情况下, |
| 415 | what we're gonna do is just load the model. | 我们要做的就是装上模型 |
| 416 | Now, when we load the model, | 现在,当我们加载模型, |
| 417 | same thing it asks us for the settings. | 同样的东西 它要求我们设置。 |
| 418 | We're gonna go ahead and load that. | 我们要继续装货 |
| 419 | It will take a second, | 需要一点时间 |
| 420 | and then we will be able to view that here | 然后我们可以在这里看到 |
| 421 | from this terminal view | 从此终端视图 |
| 422 | and also chat with it directly from the chat window. | 还直接从聊天窗口与它聊天。 |
| 423 | You can load multiple models at once, | 你可以马上装上多个模型, |
| 424 | and you'll be able to see the models that are loaded up here | 你会看到装满的模型 |
| 425 | as well as the size and then to check them. | 还有大小 然后检查一下 |
| 426 | So you can see that I'm currently using 5.58 gigabytes | 所以,你可以看到,我目前使用5.58千兆字节 |
| 427 | of the 63 gigabytes of RAM that I have. | 我拥有63千兆字节的RAM |
| 428 | However, we're talking about GPU memory here, | 但是,我们说的是GPU的记忆, |
| 429 | so that's not really 100% accurate. | 所以这不是真的100%准确。 |
| 430 | So if we go here to the terminal, | 如果我们到终点站 |
| 431 | we can now see that this model is loaded. | 我们现在可以看到这个模型已经装满了。 |
| 432 | I can view all of the API stuff for this | 我可以看所有API的东西 |
| 433 | in case I'm a developer and I wanna directly chat with it | 以防我是开发商 我想直接和它聊天 |
| 434 | using something like a curl command. | 使用类似卷曲命令。 |
| 435 | If that doesn't make sense to you, don't worry. | 如果你觉得不合理 别担心 |
| 436 | If you just wanna chat with the model, | 如果你想和模特儿谈谈 |
| 437 | you can go over to the chat view. | 你可以去聊天视图。 |
| 438 | So from here, we'll press new chat. | 因此,从这里,我们将按新的聊天。 |
| 439 | We're just gonna select the model that's already loaded. | 我们只是要选择 已经上膛的模型。 |
| 440 | 因此谷歌Gemma 4. | |
| 441 | We can modify things here, for example, | 我们可以在这里修改,比如, |
| 442 | like the system prompt if we wanna do that. | 如果我们想这么做 系统就会启动 |
| 443 | And then we can just start chatting directly with the model | 然后我们就可以直接和模型聊天了 |
| 444 | just like we would inside of something like chat GPT. | 就像我们在里面 喜欢聊天GPT。 |
| 445 | Now you can see this one is extremely fast, right? | 现在你可以看到这个非常快,对不对? |
| 446 | We're getting 120 tokens per second | 我们每秒得到120个金币 |
| 447 | because it's very small. | 因为它很小。 |
| 448 | And again, I have high memory bandwidth | 再说一遍,我的记忆带宽很高 |
| 449 | because I'm using a dedicated GPU. | 因为我在使用专用的GPU. |
| 450 | You can load multiple models | 您可以装入多个模型 |
| 451 | as long as they all fit into the memory here. | 只要他们都融入这里的记忆 |
| 452 | And then again, you can adjust all of the parameters. | 然后,你可以调整所有参数。 |
| 453 | And if you want to, you can start using them | 如果你想,你可以开始使用它们 |
| 454 | from this server, which is useful, especially for coding. | 来自此服务器, 特别是用于编码 。 |
| 455 | And you can see the full logs of everything that's gone on, | 你可以看到所有事情的完整记录 |
| 456 | tokens per second speed. | 每秒速度的符号。 |
| 457 | You get the idea. | 你得到这个想法。 |
| 458 | LM Studio is very good. | LM工作室非常不错. |
| 459 | There's a lot you can do with it. | 诸法无量. |
| 460 | And if you want a full tutorial, | 如果你想要一个完整的教程, |
| 461 | leave a comment down below and I will go into it. | 下面留点意见,我再谈 |
| 462 | So the next tool on my list here is Olama. | 我名单上的下一个工具是奥拉玛。 |
| 463 | Now this is a little bit more popular for developers | 这对开发者来说更受欢迎 |
| 464 | 它和LM Studio有很相似之处 | |
| 465 | However, it's a little bit less visual | 然而,它有点少视觉 |
| 466 | and gives you a bit less control. | 让你少点控制力 |
| 467 | Now Olama is a very popular way for downloading | 现在奥拉玛是一个非常流行的下载方式 |
| 468 | and running local models. | 并运行本地模型。 |
| 469 | In order to use it, you do need to download the tool. | 为了使用,您确实需要下载工具. |
| 470 | 所以你可以直接去olama.com | |
| 471 | and it will be available inside of your terminal | 在你的终端里可以找到它 |
| 472 | as a command. | 作为命令。 |
| 473 | So if you type Olama in your terminal once it's downloaded, | 所以,如果你打 奥拉玛在你的终端 一旦下载, |
| 474 | sorry, you should see something like this | 对不起,你应该看看这样的东西 |
| 475 | where you can launch it for all kinds of tools | 在那里你可以推出它 各种工具 |
| 476 | or you can directly chat with different models. | 或者你可以直接和不同的模型聊天。 |
| 477 | Now, as well as the terminal or CLI based tool, | 以及终端或基于CLI的工具 |
| 478 | there is also a visual tool that you can open | 还有一个可以打开的视觉工具 |
| 479 | when you download the desktop application. | 当您下载桌面应用程序时。 |
| 480 | From here again, you can launch Olama | 再来一次,你可以发射奥拉玛 |
| 481 | inside of any of these harnesses | 在任何这些绳子里面 |
| 482 | and use models that you've downloaded. | 并使用你下载过的模型。 |
| 483 | You can go into the settings, right? | 你可以进入环境,对不对? |
| 484 | Or you can actually just start chatting | 或者你可以开始聊天 |
| 485 | with different models by selecting one | 通过选择一个来使用不同的模型 |
| 486 | of the ones that you have. | 你拥有的。 |
| 487 | Now, in order to download models in Olama, | 现在 为了在奥拉玛下载模型 |
| 488 | what you're gonna do is start by finding the model | 你要做的是从找到模型开始 |
| 489 | that you want. | 说你想说的 |
| 490 | So you're gonna have to go to the Olama hub. | 所以你得去奥拉玛中心 |
| 491 | So by doing that, you can go Olama and then Models. | 因此,通过这样做,你可以去奥拉玛和模型。 |
| 492 | From here, there's a bunch of models | 从这里,有一群模特 |
| 493 | that are available for Olama. | 给奥拉玛的 |
| 494 | Same thing, you can search through them. | 同样的事情,你可以搜索 通过他们。 |
| 495 | You can ask ChatGBT to help you find one. | 你可以请ChatGBT帮助你找到一个. |
| 496 | And if you find a model that you want, | 如果你找到一个你想要的模型, |
| 497 | so actually let's go maybe NemoTron 3.5 Lightning here, | 因此,实际上让我们去 也许尼莫特龙3.5闪电在这里, |
| 498 | we can see all the different sizes. | 我们可以看到所有不同的尺寸。 |
| 499 | We now understand what B stands for | 我们现在明白B代表什么 |
| 500 | and what quantization is, right? | 而什么是量化,对不对? |
| 501 | So we can have a look at them here | 好让我们看看这里 |
| 502 | and we can pull them directly inside of Olama. | 我们可以直接把他们拉进奥拉玛 |
| 503 | So the way this works is the following. | 因此,这个方法是: |
| 504 | First, you can type Olama list. | 首先,你可以输入奥拉玛列表. |
| 505 | If you type list, this will show you all the models | 如果您输入列表, 这将显示所有模型 |
| 506 | you currently have downloaded. | 您已下载 。 |
| 507 | And then if you wanna pull a model, you'll type Olama pull | 然后,如果你想拉一个模型, 你会打奥拉玛拉 |
| 508 | and then the model ID that you found from the model hub. | 然后从模型中心找到的模型ID。 |
| 509 | This is gonna download it to your computer | 这个会下载到你的电脑上 |
| 510 | and then allow you to start using it. | 然后让你开始使用它。 |
| 511 | If you wanna run a model, you can type Olama run | 如果你想运行一个模型,你可以打 奥拉玛运行 |
| 512 | and then put the model ID. | 然后把模式ID。 |
| 513 | So I'm gonna put NemoTron 3 like this | 所以我要把尼莫特龙3这样 |
| 514 | and it will start the model. | 它会启动模型。 |
| 515 | It will load it into my computer's memory. | 它会把它装进我电脑的记忆里 |
| 516 | This is why we'll always take a second at the beginning | 这就是为什么我们总是在开始的时候花一秒钟 |
| 517 | cause it actually needs to load it. | 因为它实际上需要加载它。 |
| 518 | Then you can start chatting with it | 那你可以开始跟它聊天了 |
| 519 | directly from this view. | 直接来自此视图。 |
| 520 | So here you can see, I can type something like hello world | 所以这里你可以看到,我可以打一些东西 像你好世界 |
| 521 | and I can just directly start using this local model. | 我可以直接开始使用这个本地模型。 |
| 522 | Now, as well as that, I can do that from this terminal view. | 现在,还有,我可以从这个终端的角度来做。 |
| 523 | So if I go to, let's go NemoTron nano or something, | 所以,如果我去, 让我们去尼莫特龙纳米什么的, |
| 524 | I can type hello. | 我可以打个招呼。 |
| 525 | Same thing, we need to wait for it to be loaded | 同样的事情,我们需要等待它被装上 |
| 526 | and then it will give us a response. | 然后它将给我们一个回应。 |
| 527 | Sometimes it takes a second, especially on the first load | 有时要花点时间 尤其是第一次 |
| 528 | but you can see we get the thinking | 但你可以看到 我们得到的想法 |
| 529 | and then we get the response. | 然后我们得到回应。 |
| 530 | And if we go back here, hello, how can I assist you today? | 如果我们回到这里,你好,我有什么可以帮助你今天? |
| 531 | Now this is great, | 现在,这是伟大的, |
| 532 | but Olama will also expose all of its services on an API. | 但奥拉玛也会在API上曝光它的所有服务. |
| 533 | So for example, if I type Olama help, | 比如,如果我输入奥拉玛的帮助, |
| 534 | you're gonna see an option | 你会看到一个选择 |
| 535 | of all of the different things that you can do. | 所有不同的事情 你可以做的。 |
| 536 | As long as Olama is running, | 只要奥拉玛在跑 |
| 537 | it will actually serve all of its models | 它实际上将服务于它的所有模式 |
| 538 | available on a default port. | 在默认端口上可用。 |
| 539 | I don't remember exactly what the port is | 我不记得港口到底是什么 |
| 540 | but I believe it's something like 11,434, | 但我相信是11,434号 |
| 541 | which means that you're actually able to send curl requests | 也就是说你实际上可以发出卷曲请求 |
| 542 | and use Olama from other tools | 并使用其他工具中的Olama |
| 543 | as long as it's running in the background. | 只要它运行在背景。 |
| 544 | I'm not gonna go into a full tutorial of it | 我不会去一个完整的辅导它 |
| 545 | but if you're a developer and you understand | 但如果你是一个开发者 你明白 |
| 546 | what a REST API is, | 何等为雷斯特API, |
| 547 | Olama provides that already with access | 奥拉玛已经提供 已经进入 |
| 548 | to all of the models. | 给所有的模型。 |
| 549 | It will automatically load any model that you ask it for | 它会自动加载任何您想要的模型 |
| 550 | whenever you try to send a request to it. | 当你试图向它发出一个请求。 |
| 551 | It actually has what's called an OpenAI compatible API | 它实际上有所谓的OpenAI兼容API |
| 552 | which means that you will be able to send requests | 这意味着您将可以发送请求 |
| 553 | in the same format that you would | 格式与您将采用的格式相同 |
| 554 | 像ChatGBT或者Anthropic之类的 | |
| 555 | Anyways, that is Olama. | 总之,那是奥拉玛。 |
| 556 | Let's go to the next example. | 复次举譬. |
| 557 | So the next tool on my list here is the Docker Model Runner. | 我名单上的下一个工具是Docker模型运行器。 |
| 558 | Now this is available as an experimental feature | 现在这个是实验性的 |
| 559 | 在 Docker 桌面内。 | |
| 560 | There is a bunch of restrictions with it. | 有很多限制。 |
| 561 | However, if you are gonna be doing this | 但是,如果你要这样做 |
| 562 | 在 Linux 机器上 | |
| 563 | and you have Nvidia hardware, it works very well. | 你有Nvidia的硬件, 它的工作非常好。 |
| 564 | It can work on CPU as well, I believe, | CPU也有用,我相信, |
| 565 | however, it's extremely slow. | 然而,这是极其缓慢的. |
| 566 | So with Docker Model Runner, | 因此,与多克模型运行者, |
| 567 | if you go into Docker desktop, there is some settings | 如果您进入 Docker 桌面, 有些设置 |
| 568 | I believe you need to enable this experimental feature. | 我相信你需要启用这个实验功能。 |
| 569 | You should see this Models tab. | 你应该看看这个模型标签。 |
| 570 | From the Models tab, you'll be able to go to Docker Hub | 从模型标签,你可以去Docker枢纽 |
| 571 | and then here you can pull all of the same type of models | 然后在这里,你可以拉 所有相同的型号 |
| 572 | as you would be able to inside of like Olama or LM Studio. | 就像奥拉玛或LM工作室 |
| 573 | Once you have a model here, you can chat with it directly. | 一旦你有一个模型在这里,你可以直接与它聊天. |
| 574 | So I have Gemma 3, for example, | 所以我有Gemma 3,例如, |
| 575 | same thing, it will automatically load the model for me. | 同样的,它会自动加载模型给我。 |
| 576 | And then I can type something like hello. | 然后我可以打一个像你好的东西。 |
| 577 | Now, similarly to all of the other tools, | 现在,和所有其他工具一样, |
| 578 | this will also expose a REST API on a different port. | 这也将会在不同的端口暴露一个REST API. |
| 579 | I believe it's 12,434 or 343 | 我想是12,434或343个 |
| 580 | that will allow you the ability to chat with these models | 这样你就能和这些模型聊天 |
| 581 | without being directly inside of this interface. | 而不直接在此界面内。 |
| 582 | You can also inspect the model, | 你也可以检查模型, |
| 583 | see all the information about it, et cetera. | 见一切诸法等. |
| 584 | What's interesting about the Docker Model Runner | Docker模型运行器有什么有趣的 |
| 585 | is that it actually treats models like containers. | 实际上它把模型当成容器 |
| 586 | What that means is that you can write Docker files, | 这意味着你可以写Docker文件, |
| 587 | you can write compose files, | 你可以写作曲文件, |
| 588 | and you can actually have models shipped directly | 也可以直接运出模型 |
| 589 | with your applications and be dependencies exposed | 与您的应用程序并暴露依赖关系 |
| 590 | through Docker kind of services, | 通过多克的服务, |
| 591 | which is a lot more complicated | 这要复杂得多 |
| 592 | than I'm going to get into in this video. | 而不是我要进入这视频。 |
| 593 | But if you do use Docker and you're familiar with this | 但是如果你真的用Docker 你熟悉这个 |
| 594 | and you use it for your apps and you want local models, | 你用它来做你的应用 你需要本地的模型, |
| 595 | this is a really good way to deploy them. | 这是部署他们的好办法 |
| 596 | Now, I'll also show you that there is a CLI based tool. | 现在,我还要告诉你,有一个基于CLI的工具. |
| 597 | So similarly to what we had before, | 跟我们以前一样 |
| 598 | if I type something like Docker model, | 如果我输入的东西 像Docker模型, |
| 599 | you can see that we can configure, | 你可以看到,我们可以配置, |
| 600 | inspect, install the runner, push RM, | 检查,安装跑者,推RM, |
| 601 | view the models, load, unload. | 查看模型,装货,卸货。 |
| 602 | You guys get the idea | 你们知道这个主意 |
| 603 | and you can view models directly inside of here as well. | 也可以直接在这里查看模型 |
| 604 | So this is a really powerful feature. | 这是一个非常强大的特征。 |
| 605 | And if you want a full tutorial on it, | 如果你想要一个完整的辅导, |
| 606 | I have actually done that on my channel. | 我确实在我的频道上做了 |
| 607 | You can see the easiest way to run LM's locally, | 你可以看到最简单的方法 运行本地的LM, |
| 608 | Docker Model Runner tutorial, | Docker 模型运行器教程, |
| 609 | goes through all of the features | 通过所有的特性 |
| 610 | and we'll even show you all of the Docker files | 我们甚至会给你们看所有的Docker文件 |
| 611 | and how to set it up with the automatic deployment. | 和如何设置它与自动部署。 |
| 612 | 好吧,这是Model Runner。 | |
| 613 | Now let's move to the last one, which is full code. | 现在,让我们移动到最后一个, 这是完整的代码。 |
| 614 | Okay, so the last example I have for you | 好,我给你的最后一个例子 |
| 615 | is actually running models using just code. | 正在使用简单的代码运行模型。 |
| 616 | So this means that we're actually gonna bring | 所以这意味着我们真的会带来 |
| 617 | in our own inference engine, in this case, Llama CPP, | 在我们自己的推论引擎中, 在本案中,Llama CPP, |
| 618 | and not rely on something like Olama | 而不是依赖象奥拉玛一样的东西 |
| 619 | 或者Docker Model Runner 或者LM Studio来为我们做这个 | |
| 620 | Now, the big surprise is that Llama CPP is the engine | 现在,最大的惊喜是Llama CPP是引擎 |
| 621 | that pretty much all of the tools that we just looked at | 几乎所有的工具 我们刚刚看 |
| 622 | are already using, | 已经在使用, |
| 623 | but if we want to invoke it directly ourself, | 但如果我们想直接援引它, |
| 624 | we can do that. | 我们可以做到这一点。 |
| 625 | So for example, you'll see we have this Gwen 2.5 model, | 比如说,你会看到我们有Gwen 2.5模型 |
| 626 | which I've downloaded locally on my computer. | 我从电脑上下载的 |
| 627 | Again, this is literally just a file | 再说一遍,这只是一个文件 |
| 628 | that contains a bunch of numbers. | 里面有一堆数字 |
| 629 | Now, if I want to run this, | 现在,如果我想运行这个, |
| 630 | normally I would need a Llama or something like that, | 通常我需要拉玛之类的东西 |
| 631 | but I can actually write code | 但我可以写代码 |
| 632 | that will invoke it directly for me. | 它会直接为我引用它。 |
| 633 | So you'll see that I can just load the model. | 所以你会看到,我可以只是装上模型。 |
| 634 | I can then create a response using this package. | 然后,我可以用这个软件包创建一个响应。 |
| 635 | And if I just run the code here, | 如果我在这里运行密码 |
| 636 | you will see that I get the following. | 你会看到,我得到以下。 |
| 637 | In my heart, I run AI on my desk running free, | 在我的心里,我运行AI在我的桌子上 免费运行, |
| 638 | and it wrote me write a haiku or whatever you call this | 它写我写一个海库 或任何你所谓的这个 |
| 639 | about running AI on your own computer. | 关于运行AI 在自己的计算机。 |
| 640 | Now I can change this prompt to be, | 现在我可以改变这个速度, |
| 641 | hey, who is Tim or something? | 嘿,谁是蒂姆什么的? |
| 642 | I don't know if that's gonna give us anything meaningful, | 我不知道这会不会给我们带来什么意义 |
| 643 | but let's run this. | 但让我们运行这个。 |
| 644 | And Tim is a character from a popular video game, | 蒂姆是一个流行电子游戏的人物 |
| 645 | whatever, right? | 随便吧? |
| 646 | So this is a very small model. | 这是一个很小的模型。 |
| 647 | Of course, it's not gonna give us good responses, | 当然,它不会给我们好的答复, |
| 648 | but you get the idea. | 但你明白这个想法 |
| 649 | We can run it fully locally. | 我们可以完全在当地运行。 |
| 650 | Now, one thing to keep in mind is that as well as doing this, | 现在,有一件事要记住 除了这样做, |
| 651 | we can actually chat with models | 我们可以和模特儿聊天 |
| 652 | that are running on our own computer | 正在我们自己的电脑上运行 |
| 653 | through services like a Llama. | 像Llama这样的服务。 |
| 654 | So like I was mentioning, | 就像我刚才提到的那样 |
| 655 | if a Llama is installed and running, | 如果一个Llama安装和运行, |
| 656 | you can specify the model that's actually available | 您可以指定实际可用的模型 |
| 657 | 在你下载的Llama。 | |
| 658 | And then similarly to before, | 跟以前一样 |
| 659 | I can run something like this. | 我可以运行这样的东西。 |
| 660 | In this case, it's gonna be a bit slower | 这样的话,会慢一点 |
| 661 | because Llama 2 is much larger. | 因为Llama 2是更大的。 |
| 662 | And you'll see that we actually get the response, right? | 你会看到,我们实际上得到了回应,对不对? |
| 663 | And it says, hey, | 它说,嘿, |
| 664 | someone might choose to run an AI model locally, | 有人可能会选择在当地运行一个AI模型, |
| 665 | blah. | |
| 666 | And it's using that a Llama backend service. | 它使用Llama后端服务。 |
| 667 | And if I wanted to, | 如果我想的话, |
| 668 | I could even change this to the LLM API, right? | 我甚至可以把它改成LLM API,对不对? |
| 669 | 或 LM Studio API 或 Docker 模型运行器 API | |
| 670 | and do the exact same thing right from code. | 并做同样的事情 从代码。 |
| 671 | So this is kind of the more manual method, | 所以,这是一种 更手工的方法, |
| 672 | but most developers are gonna end up managing their models | 但大多数开发者最终会管理他们的模型 |
| 673 | through something like a Llama. | 穿过像Llama的东西。 |
| 674 | And then invoking them in code using a method like this. | 然后用这种方法在代码中引用它们。 |
| 675 | Okay, so that wraps up the demos. | 好吧,这样就结束了演示。 |
| 676 | Now let's talk about which method you should actually use. | 现在我们来谈谈你应该使用哪种方法。 |
| 677 | So here's my honest take. | 所以,这是我诚实的拍摄。 |
| 678 | If you just wanna chat with the model | 如果你想和模特儿谈谈 |
| 679 | and you never wanna see a terminal, | 你从不希望看到一个终端, |
| 680 | then you can use something like LM Studio. | 这样你就可以用LM工作室了 |
| 681 | It's genuinely one of the easiest ways to download models | 这是最容易下载模型的方法之一 |
| 682 | and has some of the most amount of features. | 并且拥有一些数量最多的特征. |
| 683 | If you're a developer and you want a model | 如果你是开发商 想要一个模型 |
| 684 | running on your own machine | 在自己的机器上运行 |
| 685 | that your scripts and apps can talk to, | 你的剧本和应用程序可以说话, |
| 686 | definitely use a Llama. | 绝对要用拉玛 |
| 687 | That's what I reach for most days | 大部分时间我都是这样 |
| 688 | and it works really well locally. | 并且它在当地非常有效。 |
| 689 | If you're already living inside of a Docker container | 如果你已经住在多克容器里 |
| 690 | and you want models sitting in that stack right next to you, | 你想让模特坐在你旁边 |
| 691 | 然后使用Docker Model Runner,对吗? | |
| 692 | It's really good in production | 生产真的很好 |
| 693 | if you're actually building and deploying things out. | 如果你真的在建造和部署东西。 |
| 694 | And lastly, if you wanna understand | 最后,如果你想了解的话 |
| 695 | everything that you're doing and run models yourself | 你所做的一切,你自己经营模特 |
| 696 | in probably the most efficient way, | 也许是最有效的方式, |
| 697 | then you can use your own code to do so. | 那样你就可以用你自己的代码了 |
| 698 | Of course, you don't need to use Python. | 当然,你不需要使用Python。 |
| 699 | This is just a quick example. | 这只是一个简单的例子。 |
| 700 | And with that in mind, just remember | 记住这一点,只要记住 |
| 701 | that all of these tools at the end of the day | 最后这些工具 |
| 702 | are using the same building blocks | 使用相同的组件 |
| 703 | that we talked about earlier. | 我们之前说过的 |
| 704 | They have a model, which again, | 他们有一个模型,再次, |
| 705 | is literally just a bunch of numbers in a file | 字面上只是文件里的一堆数字 |
| 706 | and they have an inference engine | 他们有一个推论引擎, |
| 707 | and a bunch of other fancy features on top of it. | 上面还有一堆其他的花样 |
| 708 | If you understand that, you understand local models | 如果你明白这一点,你就明白本地的模型 |
| 709 | and hopefully this video helped get you off the ground | 希望这段视频能帮助你脱身 |
| 710 | and running your first one on your own device. | 然后用自己的设备运行第一个 |
| 711 | Anyways, guys, that's all that I have for you. | 总之,伙计们,我只有这些了 |
| 712 | If you enjoyed, make sure to leave a like, subscribe, | 如果你喜欢, 确保离开类似,订阅, |
| 713 | and I will see you in the next one. | 我会看到你在下一个。 |
| 714 | Thanks for watching. | 谢谢你的观看。 |
| 715 | Bye. | 再见。 |
该视频共有字幕 715 条。解锁更多字幕为会员功能，请移动到 价格
P2 · 【本地AI】手把手教你在电脑上跑AI模型 · 零基础指南 p02 英文字幕 (P2)
字幕摘录
| 时间 | 英文 | 中文 |
|---|---|---|
| 0:00 | Everybody is talking about running AI locally, | 大家都在谈论本地的人工智能 |
| 0:03 | but almost nobody explains what it is | 但几乎没人解释是什么 |
| 0:05 | or how you actually get it to work. | 或者你如何真正让它工作。 |
| 0:07 | Now you get these videos where people throw around terms | 现在,你得到这些视频 人们扔周围的条件 |
| 0:09 | 如重量或量化、VRAM、GGUF、 | |
| 0:13 | and it sounds like you need a PhD | 听起来你需要博士学位 |
| 0:15 | and a $10,000 computer just to try this out. | 和一万块的电脑 只是试试这个。 |
| 0:18 | And that's exactly why most people give up | 这就是为什么大多数人放弃 |
| 0:20 | and just keep paying for subscriptions like ChatGPT. | 继续支付像ChatGPT这样的订阅费. |
| 0:23 | So let me give you the honest one sentence version here. | 所以,让我给你一个诚实的句子。 |
展开字幕全文（715 条）
| 序号 | 英文 | 中文 |
|---|---|---|
| 1 | Everybody is talking about running AI locally, | 大家都在谈论本地的人工智能 |
| 2 | but almost nobody explains what it is | 但几乎没人解释是什么 |
| 3 | or how you actually get it to work. | 或者你如何真正让它工作。 |
| 4 | Now you get these videos where people throw around terms | 现在,你得到这些视频 人们扔周围的条件 |
| 5 | 如重量或量化、VRAM、GGUF、 | |
| 6 | and it sounds like you need a PhD | 听起来你需要博士学位 |
| 7 | and a $10,000 computer just to try this out. | 和一万块的电脑 只是试试这个。 |
| 8 | And that's exactly why most people give up | 这就是为什么大多数人放弃 |
| 9 | and just keep paying for subscriptions like ChatGPT. | 继续支付像ChatGPT这样的订阅费. |
| 10 | So let me give you the honest one sentence version here. | 所以,让我给你一个诚实的句子。 |
| 11 | That is that local AI is simply a model file | 这就是说本地的AI只是一个模型文件 |
| 12 | that's sitting on your computer and a program that runs it. | 坐在你的电脑和运行程序上 |
| 13 | That's it. | 就这样 |
| 14 | No cloud, no API keys, no internet, no subscription. | 没有云,没有API键,没有互联网,没有订阅. |
| 15 | Everything else is really just a small detail. | 其他一切都只是个小细节 |
| 16 | So in this video, I'm gonna break down | 因此,在这个视频中,我要打破 |
| 17 | what's actually happening under the hood, | 引擎盖下到底发生了什么 |
| 18 | the real building blocks, and none of the fluff. | 真正的建筑构件, 和没有绒毛。 |
| 19 | And then I'm gonna show you how to run a model | 然后我会告诉你如何运行一个模型 |
| 20 | on your own machine in four completely different ways, | 在你自己的机器上 四种完全不同的方式, |
| 21 | from an app that you download | 从您下载的应用程序中 |
| 22 | all the way to pure low-level code | 一直到纯低层代码 |
| 23 | where you're running it yourself. | 在那里你运行它自己。 |
| 24 | Now by the end of this video, | 现在,在这个视频结束时, |
| 25 | you're gonna understand local AI | 你会明白本地的AI |
| 26 | better than most people that are posting about it. | 比大多数贴报的人要好 |
| 27 | So let's dive in. | 所以让我们潜入。 |
| 28 | So first, let's clear up | 先说清楚 |
| 29 | the most common confusion that I see. | 我所看到的最常见的混乱 |
| 30 | Now that's what's actually different | 这才是真正的不同 |
| 31 | between something like ChatGPT and a local model. | 在像ChatGPT之类的东西 和本地模型之间 |
| 32 | Now when you use ChatGPT, Cloud, Gemini, | 当你使用ChatGPT,云,双子座, |
| 33 | really any of these tools, | 任何这些工具, |
| 34 | what's happening is that you type a message, | 发生的事情是 你输入一个消息, |
| 35 | that message leaves your computer, | 留言留下你的电脑, |
| 36 | it travels over the internet to something like a data center | 它在互联网上游历到一个像数据中心的东西 |
| 37 | where a massive computer that you don't own | 你没有的大型电脑 |
| 38 | runs a giant model, | 运行一个巨大的模型, |
| 39 | and then the answer gets streamed back to your screen. | 然后答案被流回你的屏幕。 |
| 40 | So your computer did basically nothing. | 所以,你的电脑基本上什么都没做。 |
| 41 | It's really just a window into somebody else's machine. | 它真的只是一个窗口 进入别人的机器。 |
| 42 | Now local AI completely flips that. | 现在本地的AI完全推翻了这一点. |
| 43 | The model, so the actual file | 模型,所以实际文件 |
| 44 | that contains all of the intelligence | 包含所有情报 |
| 45 | gets downloaded onto your computer. | 下载到电脑上 |
| 46 | And when you ask it a question, | 当你问这个问题时, |
| 47 | your own CPU or GPU is the one that's doing the work. | 您的CPU或GPU是 做这项工作。 |
| 48 | So nothing leaves your machine, | 所以,没有什么离开你的机器, |
| 49 | and that gives you three big advantages. | 这给你三个大优势 |
| 50 | Now first, it's private. | 现在,首先,这是私人的。 |
| 51 | That's because your data never goes anywhere, | 那是因为你的数据从不消失 |
| 52 | it stays on your computer. | 它留在你的电脑上。 |
| 53 | Second, it's free because there's no subscription | 第二,免费,因为没有订户 |
| 54 | and there's no per-token cost. | 而且没有每桶成本。 |
| 55 | And lastly, it works offline. | 最后,它的工作离线。 |
| 56 | So if you're in a plane or a coffee shop with terrible Wi-Fi, | 所以,如果你在飞机或咖啡店 用可怕的无线, |
| 57 | doesn't matter, you can use these local models. | 无所谓,你可以使用这些本地模型。 |
| 58 | Now I do wanna be honest here about the trade-offs, | 现在,我想诚实地在这里 关于权衡, |
| 59 | which is that the models that you're gonna be able | 也就是你能够做的模型 |
| 60 | to run at home are much smaller than the frontier models | 在家跑步比边疆模式要小得多 |
| 61 | that you'd use from something like Cloud or OpenAI. | 你会用在云或OpenAI之类的东西上 |
| 62 | But they have gone shockingly good | 但是,他们已经走了惊人的好 |
| 63 | over the last couple of years, | 在过去几年里, |
| 64 | and for a huge amount of everyday tasks, | 和大量日常工作, |
| 65 | even coding tasks, they're more than enough. | 甚至编码任务,它们都足够了。 |
| 66 | So with that said, let's have a look | 所以说,让我们来看看 |
| 67 | at the actual pieces here, | 在实际的作品在这里, |
| 68 | so you understand local models much deeper. | 所以你更了解当地的模型 |
| 69 | Now the first part here is the model itself. | 现在第一部分是模型本身。 |
| 70 | And I wanna be really clear about what a model actually is, | 我想说清楚什么是模特儿 |
| 71 | because this is where some people imagine | 因为这就是一些人想象的 |
| 72 | some kind of magic black box. | 某种魔力黑盒。 |
| 73 | Now a model is literally just a file. | 现在一个模型其实只是一个文件。 |
| 74 | It's a really big file, could be hundreds of gigabytes, | 这是一个非常大的文件,可能是数百千兆字节, |
| 75 | that's full of numbers that we call weights. | 我们称之为重量数 |
| 76 | Now these are billions of numbers | 这些是数十亿的数字 |
| 77 | that got baked in when the model was trained. | 当模特被训练时 被烤熟了 |
| 78 | Now that file doesn't think, it doesn't run, | 现在那个文件不思考,它不运行, |
| 79 | it just sits there on your disk | 它只是坐在你的磁盘上 |
| 80 | like any other file that you would have. | 就像其他文件一样 |
| 81 | And companies like Meta, Google, Alibaba, Mistral, whatever, | 还有梅塔、谷歌、阿里巴巴、米斯特拉尔等公司 |
| 82 | they release these files for free. | 他们免费发布这些文件。 |
| 83 | Now those are your open models | 这些是你的公开模型 |
| 84 | and things that you keep hearing about | 和你们所听到的 |
| 85 | 比如Llama或者Gemma或者Quen或者DeepSeq,你懂这个意思. | |
| 86 | You can literally just download them | 你可以直接下载 |
| 87 | because they are literally just a file | 因为从字面上来说,它们只是一个文件 |
| 88 | full of billions of different numbers. | 充满了数十亿个不同的数字。 |
| 89 | Now when you go to look at these models, | 当你去看这些模型时, |
| 90 | you're gonna see names like four billion, | 你会看到40亿的名字, |
| 91 | eight billion, 70 billion. | 八十亿七百亿. |
| 92 | Now that B is billions, | 现在B是亿, |
| 93 | and it stands for the number of parameters. | 它代表参数的数量。 |
| 94 | Now that's effectively just how many numbers | 现在,这是有效的 只是多少数字 |
| 95 | are inside of that file. | 里面的档案。 |
| 96 | And the rule here is pretty simple. | 这里的规则很简单。 |
| 97 | Generally speaking, more parameters means a smarter model, | 一般来说,更多的参数意味着一个更聪明的模型, |
| 98 | but it also means a bigger file | 但它也意味着一个更大的文件 |
| 99 | that needs more memory and compute to run. | 这需要更多的记忆和计算运行。 |
| 100 | So if you look at an eight billion model, | 所以如果你看看80亿的模型 |
| 101 | this is maybe a few gigabytes in storage. | 这也许是几千兆字节的存储。 |
| 102 | Well, a 70 billion model is a file that's so large | 嗯,一个700亿的模型是一个文件 这是那么大 |
| 103 | that most laptops simply cannot even load it, | 大部分笔记本电脑都装不下 |
| 104 | and some may not even have enough storage to download it. | 而有些人甚至可能没有足够的存储器可以下载. |
| 105 | Now that leads directly into the biggest trick | 现在,这直接 进入最大的诡计 |
| 106 | that happens with local AI. | 这发生在本地AI。 |
| 107 | Now this is something called quantization. | 这是所谓的量化。 |
| 108 | Now it sounds scary and really complicated, | 现在听起来很可怕,很复杂 |
| 109 | but it's really the same idea as compressing a photo. | 但它真的相同的想法 压缩一张照片。 |
| 110 | So you can take these billions of numbers | 这样你就可以拿这些亿万的数字 |
| 111 | and you can store them with less precision. | 并且可以更精确地储存它们。 |
| 112 | And that means that this model file | 这意味着这个模型文件 |
| 113 | is gonna get dramatically smaller | 就会变小 |
| 114 | with barely any quality loss at all. | 几乎没有质量损失。 |
| 115 | So a model that would normally need 16 gigabytes of memory | 所以这个模型通常需要16千兆字节的内存 |
| 116 | in its original form might only need five, six, | 原型可能只需要五,六, |
| 117 | or seven gigabytes after quantization. | 或七千兆字节后 |
| 118 | And when you see the term gguf floating around, | 当你看到这个术语漂浮, |
| 119 | that's basically just the standard file format | 这基本上只是标准文件格式 |
| 120 | for these compressed models. | 这些压缩模型。 |
| 121 | Now this is the entire reason that normal computers | 这才是正常电脑的原因 |
| 122 | can even run AI models at all, | 甚至可以运行AI模型, |
| 123 | or at least some of the bigger ones. | 或至少一些较大的。 |
| 124 | So when you hear about quantization | 所以,当你听到关于量化 |
| 125 | or a quantized model, | 或被量化的模型, |
| 126 | just think of that as a compressed model | 把这当成一个压缩模型 |
| 127 | that's meant to make it smaller so it's easier to run. | 意思是让它变小 所以它更容易运行。 |
| 128 | When you compress these models, | 当你压缩这些模型时, |
| 129 | you keep almost the exact same performance, | 你保持几乎完全相同的表现, |
| 130 | but again, you just reduce the size drastically, | 再说一次,你只是大大缩小了尺寸 |
| 131 | so it's a lot easier to actually run. | 因此,这很容易真正运行。 |
| 132 | Now the next piece is the inference engine. | 现在下一个是推论引擎。 |
| 133 | And this is the part that almost nobody explains. | 这是几乎没人解释的部分。 |
| 134 | Remember that the model is just a file full of numbers, | 记住,模型只是一个文件 充满数字, |
| 135 | and a file can't run itself. | 并且文件不能自己运行。 |
| 136 | So you need a program | 所以你需要一个程序 |
| 137 | that can actually load these numbers into memory | 可以将这些数字装入内存 |
| 138 | and do the actual math to make the model work | 并做实际的数学 使模型工作 |
| 139 | and predict the next token. | 并预言下一个迹象。 |
| 140 | Now that program is called an inference engine. | 现在这个程序叫做推论引擎. |
| 141 | Now the most famous one is called llama.cpp. | 现在最著名的一个叫做lama.cpp. |
| 142 | And here's the secret, | 这是秘密 |
| 143 | almost every tool that I'm gonna show you today, | 几乎每一个工具 我今天要给你看, |
| 144 | so LM Studio, Olamma, Docker Model Runner, | 因此LM工作室,奥拉姆马,多克模型运行器, |
| 145 | they're all basically just a wrapper | 他们基本上都只是一个包装器 |
| 146 | around engines like this. | 绕着这种引擎 |
| 147 | Now the engine does all of the work, | 现在引擎完成了所有的工作, |
| 148 | and the tool just kind of makes it nice to use, | 而工具只是 那种使它使用, |
| 149 | which you're gonna see later on. | 稍后你会看到的。 |
| 150 | Now the last building block here is your hardware, | 最后一块是你们的硬件 |
| 151 | because there's really only one question | 因为只有一个问题 |
| 152 | that decides what local models | 决定何种地方模式 |
| 153 | you're going to be able to run. | 你会跑的 |
| 154 | Now that is, how much memory do you have on your computer, | 你电脑里有多少记忆 |
| 155 | and how fast is that memory? | 记忆有多快? |
| 156 | Now on a PC with a graphics card, | 现在在一台带显卡的PC上 |
| 157 | so a dedicated graphics card, | 所以一个专用的显卡, |
| 158 | that number is gonna be your VRAM. | 这个号码会是你的VRAM。 |
| 159 | So if you have an Nvidia 3090 or 4090 or 5090, | 所以,如果你有一个Nvidia 3090或4090或5090, |
| 160 | or I'm just naming random GPUs, | 或者我只是命名随机GPU, |
| 161 | you're gonna look at the VRAM on that device. | 你看看那个装置上的VRAM |
| 162 | Now if you're running on a modern Mac computer, | 如果你在现代Mac电脑上运行 |
| 163 | 就像M3或M4一样 任何M系列的MacBook | |
| 164 | then you're just gonna be looking at the amount of RAM | 那你就只能看看内存的数量 |
| 165 | that your computer has. | 你的电脑有。 |
| 166 | And that's because Apple shares its memory with the GPU, | 这是因为苹果公司与GPU共享其记忆, |
| 167 | and has something called unified memory. | 并有所谓的统一记忆。 |
| 168 | Now other devices have different specifications, | 现在其他设备有不同的规格, |
| 169 | but generally speaking, | 但总的来说, |
| 170 | if you have a relatively new computer, | 如果你有一个相对新的计算机, |
| 171 | if it has a dedicated graphics card, | 如果它有一个专用的图形卡, |
| 172 | 你看着VRAM。 | |
| 173 | That's the amount of memory you're gonna have | 这就是你的记忆量 |
| 174 | for running local models. | 用于运行本地模型。 |
| 175 | And if you're on a Mac, again, a modern one, | 如果你在Mac上 又是现代的 |
| 176 | you're looking at the amount of unified memory. | 你在看统一记忆的量 |
| 177 | And the rule of thumb here is pretty simple. | 这里的拇指规则很简单。 |
| 178 | The model file needs to be able to fit | 模型文件需要适合 |
| 179 | inside of that memory that you have | 在你的记忆中 |
| 180 | with a little bit of room to spare. | 还有一点空位 |
| 181 | So roughly speaking, | 所以说, |
| 182 | if you have eight gigabytes of VRAM or unified memory, | 如果你有8千兆字节 VRAM 或统一的内存, |
| 183 | you're gonna be able to run three to four billion | 你会跑到三四十亿 |
| 184 | parameter models, even without being quantized. | 参数模型,即使没有被量化. |
| 185 | And if you have 16 gigabytes of memory, | 如果你有16千兆字节的记忆 |
| 186 | you can go up to seven or eight billion parameter models. | 可以达到70或80亿个参数模型。 |
| 187 | And then if you go up to 32 gigabytes, | 然后如果你升到32千兆字节 |
| 188 | you start to be able to get into the 14 | 你开始可以进入14 |
| 189 | to 30 billion parameter range. | 到300亿参数范围。 |
| 190 | And this is where things start to feel genuinely smart, | 而这就是事情开始 感觉真正聪明, |
| 191 | especially for local models. | 特别是当地模特儿 |
| 192 | But keep in mind, | 但记住, |
| 193 | you don't need a monster computer to do any of this. | 你不需要怪物电脑来做这些 |
| 194 | Even with something like your phone, | 即使是用你的手机一样的东西 |
| 195 | you can run small models already. | 你可以运行小模型了。 |
| 196 | And one thing that carries over from cloud AI | 还有从云中传来的一件事 |
| 197 | is that the context windows | 是上下文窗口 |
| 198 | to the model short-term working memory | 到模式短期工作记忆 |
| 199 | is going to actually affect the amount of space | 将会影响空间的大小 |
| 200 | that's being taken up. | 这正在处理中。 |
| 201 | So if you have long conversations, bigger documents, et cetera, | 所以,如果你有长话短说, 更大的文件等等, |
| 202 | that's also going to fill up your memory. | 那也会让你的记忆充满 |
| 203 | Now I'm gonna explain this a little bit more in detail | 现在我要详细解释一下 |
| 204 | because this is super important. | 因为这很重要 |
| 205 | But the basic idea is that the one number | 但基本想法是,一个数字 |
| 206 | you need to be aware of is how much memory, | 你需要知道 有多少记忆, |
| 207 | either VRAM or unified memory is on your computer. | 无论是VRAM还是统一的内存都在你的电脑上。 |
| 208 | When you look at a model, whether it's quantized or not, | 当你看一个模型, 无论它是否被量化, |
| 209 | it needs to be able to fit comfortably | 它需要能够舒适地装配 |
| 210 | within that memory range. | 在那个记忆范围内 |
| 211 | So if you have 32 gigabytes of VRAM | 所以,如果你有32千兆字节 VRAM |
| 212 | and a model is 25 gigs, that's fine. | 一个模型是25个工作,没关系。 |
| 213 | You'll be able to run it, right? | 你会运行它,对不对? |
| 214 | However, one thing to keep in mind with memory | 不过,有一件事要记住 |
| 215 | is also the speed of the memory. | 也是内存的速度。 |
| 216 | So while you will be able to run much larger models | 所以当你可以运行 更大的模型 |
| 217 | on things like modern Macs that have 128 gigabytes | 在像现代Macs这样的 有128千兆字节的东西上 |
| 218 | of unified memory, the inference speed of those models | 这些模型的推论速度 |
| 219 | is gonna be much slower than a similar model | 比一个类似的模型慢得多 |
| 220 | that's running on something like a dedicated GPU | 运行在像一个专用GPU的东西上 |
| 221 | 或Nvidia RTX GPU. (中文(简体) ). | |
| 222 | The reason for this is the speed of the memory. | 其原因是记忆的速度. |
| 223 | So while the memory will dictate the size of the model | 所以虽然记忆会决定模型的大小 |
| 224 | and the more memory you have, | 越多的记忆, |
| 225 | the smarter models you can run, | 你能够运行的更聪明的模型, |
| 226 | the memory speed matters for the tokens per second | 每秒符号的内存速度很重要 |
| 227 | and the inference speed that you're going to be able | 和推论速度 你将有能力 |
| 228 | to generate. | 生成。 |
| 229 | There's a lot of different techniques here | 这里有很多不同的技术 |
| 230 | and things that I could get into, | 和我可以进入的东西, |
| 231 | but generally speaking, | 但总的来说, |
| 232 | if we talk about dedicated graphics cards, | 如果我们谈论专用显卡, |
| 233 | again, typically in the Nvidia family, | 通常在恩维迪亚家族, |
| 234 | these are much faster, sometimes two or three times faster | 速度快得多 有时快两三倍 |
| 235 | at inference speed, but they usually have less capacity. | 在推论速度上, 但他们通常能力较低。 |
| 236 | So for example, I have 24 gigabytes of VRAM in my 4090 | 比如,我4090年有24千兆字节的VRAM |
| 237 | and it's very fast and can generate 200 tokens per second | 它的速度很快,每秒能产生200个令牌 |
| 238 | for some of the models that I run. | 我运行的一些模型。 |
| 239 | However, I can't run models that are 70 billion parameters | 但我不能运行700亿参数的模型 |
| 240 | like I might be able to on my Mac. | 碞钩ии沉 |
| 241 | However, on my Mac, those models are really slow | 然而,在我的Mac上,这些模型真的很慢 |
| 242 | because the memory speed is significantly slower. | 因为内存速度明显较慢. |
| 243 | So you're going to be looking at memory speed | 所以你要看记忆速度 |
| 244 | as well as memory capacity. | 以及记忆能力。 |
| 245 | Those are the two things that are going to dictate | 这些是两件事要决定的 |
| 246 | what you can do with local AI. | 你对本地的AI能做什么? |
| 247 | And there's always going to be a trade-off | 总是有取舍 |
| 248 | in terms of the size of the model | 从模型的大小来看 |
| 249 | and the inference speeds that you're getting. | 和你的推论速度。 |
| 250 | Typically smaller models are going to be much faster. | 通常较小的模型会更快 |
| 251 | And again, the most important thing | 再说一遍,最重要的是 |
| 252 | is that whatever model you run, | 是无论你运行的模型, |
| 253 | it needs to fit in this memory | 它需要融入这个记忆 |
| 254 | while it still can run if it's not in there, | 如果里面没有它 还能运行 |
| 255 | it's going to be so slow that it's practically unusable. | 速度会太慢 以至于无法使用 |
| 256 | Generally speaking, if you're looking at models | 一般来说,如果你在看模特儿 |
| 257 | between 14 and 35 billion parameters, | 140亿至350亿参数之间, |
| 258 | those are going to be a really good sweet spot. | 这将是一个非常好的甜点。 |
| 259 | They're going to give you pretty decent performance | 他们会给你一个很好的表演 |
| 260 | and you're not going to feel | 你不会觉得 |
| 261 | like you're missing out on too much. | 就像你错过了太多。 |
| 262 | If you go up to huge models, | 如果你去大模型, |
| 263 | like 120 billion parameters, 250 billion parameters, | 比如1200亿参数,2500亿参数, |
| 264 | you're talking about needing extremely high-end hardware | 你说的是需要非常高端的硬件 |
| 265 | running at slow inference speeds. | 以缓慢推论速度运行。 |
| 266 | And it's very difficult to actually run those | 很难真正运行这些 |
| 267 | at scale on your own machine. | 在自己的机器上。 |
| 268 | Anyways, with that in mind, let's keep going here. | 无论如何,考虑到这一点, 让我们继续在这里。 |
| 269 | I want to tell you about something really interesting. | 我想告诉你一些有趣的事 |
| 270 | Now the whole reason you'd even bother | 现在的整个原因 你甚至会烦恼 |
| 271 | running models locally comes down to one main thing, | 本地运行的模型 归根结底是一件事, |
| 272 | which I know you all want, which is control. | 我知道你们都想要的 这就是控制 |
| 273 | You pick the model, you own the setup | 你选模特儿,你自有一套 |
| 274 | and nobody can change the deal in. | 没有人可以改变交易。 |
| 275 | But here's the problem, right? | 但问题是,对吧? |
| 276 | Almost every tool that you'd actually want to use | 几乎每个工具 你真正想要使用 |
| 277 | locks you into one provider's model. | 把你锁在一个供应商的模型里 |
| 278 | Now that's exactly what Mindhub Cowork, | 这正是明德胡布同事 |
| 279 | who's the sponsor of today's video, is built to fix. | 谁是今天录影带的赞助商 是用来修复的 |
| 280 | Now it's open source, free to use, | 现在它是开源,自由使用, |
| 281 | and it has a real model router. | 它有一个真正的模型路由器。 |
| 282 | 你可以经营Claw、GPT、Chaminade、Jeepseq | |
| 283 | or the same local models that I've been talking about | 或者我一直在谈论的本地模特儿 |
| 284 | right here, all inside of this workspace. | 在这里,所有在这个工作空间内部。 |
| 285 | So the workflow is simple. | 所以工作流程很简单。 |
| 286 | You brief the built-in agent harness, which is Anton, | 你向内置特工介绍一下 也就是安东 |
| 287 | walk away and come back to finished work. | 走开,回来完成工作。 |
| 288 | Now I asked it to research the latest coding models | 现在我要求它研究最新的编码模型 |
| 289 | and build me a comparison dashboard. | 帮我造一个比较仪表板 |
| 290 | And this is what it came back with, | 这就是它回来的原因 |
| 291 | an actual dashboard that I can open and share, | 一个真正的仪表板 我可以打开和分享, |
| 292 | not just a wall of text. | 不只是文字墙。 |
| 293 | And this is the part that connects | 这是连接的部分 |
| 294 | to everything in this video. | 给这段录像里的所有内容 |
| 295 | When a better model drops, | 当一个更好的模型下降, |
| 296 | whether it's local or a cloud model, | 无论是本地还是云型 |
| 297 | I can swap it in and I don't have to change anything. | 我可以把它换进去,我不需要改变任何东西。 |
| 298 | So I have the same workspace, the same work, | 所以我有相同的工作空间,相同的工作, |
| 299 | and since it's fully open source, | 既然它完全开源 |
| 300 | you can clone the repo, | 你可以复制Repo, |
| 301 | spin it up locally in just a few commands, | 转动到局部 仅几个命令, |
| 302 | or just download the dedicated Mac or Windows app. | 或者只下载专用的Mac或Windows应用程序. |
| 303 | The whole thing runs on your own hardware, | 整个事情运行在你的硬件上, |
| 304 | which is basically the end game | 这基本上是结束游戏 |
| 305 | of what we're testing today. | 今天的测试 |
| 306 | Now they also have a hosted version, | 现在,他们还有一个主机版本, |
| 307 | but honestly, I love the desktop app | 但老实说,我喜欢桌面应用 |
| 308 | as it's very easy to use. | 因为它很容易使用。 |
| 309 | So I'm gonna leave a link to it in the description. | 所以我要在描述中留下一个链接。 |
| 310 | Try it out. | 试试看 |
| 311 | Again, it really goes nicely with these local models. | 再一次,它真的 与这些当地模型。 |
| 312 | And now let me show you | 现在让我告诉你 |
| 313 | how we can actually run local models. | 如何实际运行本地模型。 |
| 314 | So there's a lot of ways to run a local model. | 因此当地模式的运行方式很多. |
| 315 | And just like anything in software, | 就像软件里的任何东西一样 |
| 316 | it really comes down to how much control you want. | 这真的取决于你想控制多少 |
| 317 | So I've broken this into four different tiers. | 所以我把它分为四层。 |
| 318 | Now at the top, we've got LM Studio. | 现在最上面的是LM工作室 |
| 319 | This is a regular desktop app. | 这是普通的桌面应用程序 。 |
| 320 | You can click, you can download, you can touch everything, | 你可以点击,你可以下载, 你可以触摸一切, |
| 321 | and you don't really need to go into the terminal. | 而你真的不需要进入终端。 |
| 322 | Now below that, we have Olamo. | 下面是奥拉莫 |
| 323 | This is a really popular option, especially for developers, | 这是一个非常流行的选择, 特别是对开发者来说, |
| 324 | and it's just one command inside of your terminal | 它只是一个命令 在你的终端内部 |
| 325 | where you can talk with models, spin up a local server. | 您可以与模型交谈, 旋转本地服务器。 |
| 326 | You get the idea. | 你得到这个想法。 |
| 327 | 然后我们有了Docker Model Runner。 | |
| 328 | Now this is really good | 现在,这是真的好 |
| 329 | because it treats models like containers, | 因为它对待模型 像容器, |
| 330 | which is perfect if you're actually gonna be deploying | 如果你真的要部署,那就完美了 |
| 331 | these alongside live applications. | 它们与现场应用同时存在。 |
| 332 | And at the bottom of my list here, we have full code. | 在我的名单底部, 我们有完整的代码。 |
| 333 | This is where you're running a model in pure Python | 这就是你运行一个模型 在纯Python |
| 334 | and you see every single piece. | 你可以看到每一块。 |
| 335 | Now, no matter which way you wanna run these models here, | 现在,不管你想用哪种方式运行这些模型, |
| 336 | you're gonna be making three decisions. | 你会做出三个决定 |
| 337 | You're gonna pick a model, | 你要选一个模特儿 |
| 338 | you're gonna pick a size and quantization | 你会选择一个大小和量化 |
| 339 | that fits inside of your memory, | 这符合你的记忆, |
| 340 | and you're gonna decide how you wanna talk to it. | 你要决定怎么跟它说话 |
| 341 | So whether that's a chat window or something like code. | 所以,无论这是一个聊天窗口 还是类似代码的东西。 |
| 342 | If you keep that in mind, | 如果你记住这一点, |
| 343 | every single one of these tools is gonna make sense. | 这些工具中的每一个都会有意义 |
| 344 | So let's start at the top. | 故从上起. |
| 345 | And again, I'm gonna go through all of them | 再说一遍,我要翻遍他们所有 |
| 346 | and show you exactly how to run local models. | 告诉你如何运行本地的模型。 |
| 347 | Let's dive in. | 让我们潜入。 |
| 348 | So the first tool on my list here is LM Studio. | 我的第一个工具是LM工作室。 |
| 349 | I'll give you a quick walkthrough, | 我给你一个快速的走过, |
| 350 | but this is one of the best ways to run local models. | 但这是管理本地模型的最佳方法之一 |
| 351 | Now, once you download the tool again, it's completely free, | 现在,一旦你再次下载工具, 它是完全免费的, |
| 352 | you're gonna be brought into a view | 你会被带入风景 |
| 353 | that looks something like this. | 看起来像这样 |
| 354 | From here, you're gonna go into the model view, | 从这里开始,你会进入模型视图, |
| 355 | looks like kind of a robot icon on the left-hand side, | 看起来像是左边的机器人图标 |
| 356 | and you'll be able to search | 你就可以去寻找 |
| 357 | through all of the available models | 通过所有可用的模式 |
| 358 | that you can download directly here. | 您可以在此直接下载。 |
| 359 | Now, what you'll wanna do is search for a model | 现在,你想做的是寻找一个模型 |
| 360 | that matches the kind of relative size | 匹配相对大小的类型 |
| 361 | or amount of memory that you have. | 或你的记忆量 |
| 362 | Again, if we're talking about larger high-end machines, | 再说一遍 如果我们说的是更大的高端机器 |
| 363 | you can typically get away with 27 billion, 35, | 你通常能逃出270亿 35亿 |
| 364 | 30 billion parameters, et cetera. | 三百亿参数等. |
| 365 | If you have eight gigs of RAM or 16 gigs of RAM, | 如果你有8个工作 或16个工作 RAM, |
| 366 | look for ones that are eight billion parameters | 寻找80亿的参数 |
| 367 | or four billion parameters, much smaller sizes. | 或40亿参数,大小小得多。 |
| 368 | So for example, we have Gwen 3.8. | 例如,我们有格温3.8。 |
| 369 | You'll notice that if I click on this, | 你会注意到,如果我点击这个, |
| 370 | I can view different levels of quantization | 我可以看到不同层次的量化 |
| 371 | and see the change in size right here. | 看这里大小的变化 |
| 372 | You also see kind of some icons or indications | 您还看到一些图标或指示 |
| 373 | of which model is the best for you based on your hardware. | 根据你的硬件,哪个型号最适合你。 |
| 374 | So you can see when it says full GPU offload possible | 所以你可以看到,当它说全GPU卸载的可能性 |
| 375 | and a little thumbs up, | 并竖起一小拇指, |
| 376 | that's the one that you would wanna go with. | 那是你想去的那个 |
| 377 | And notice this number of Q4, | 注意这个Q4的号码 |
| 378 | that's the level of precision or the quantization level. | 这是精确度或量化度。 |
| 379 | So the lower, so like Q4, Q2, Q1, | 所以越低,就像Q4,Q2,Q1, |
| 380 | the more quantization you have, right? | 越多的量化,对不对? |
| 381 | So if you have Q8, you can see this is bigger | 所以,如果你有Q8,你可以看到 这是更大的 |
| 382 | than Q6 or Q4, and it's a pretty drastic difference. | 比Q6或Q4, 这是一个相当巨大的差异。 |
| 383 | You're also gonna wanna look at the capabilities, | 你也想看看他们的能力 |
| 384 | so vision, tool use, reasoning. | 因此,视觉,工具的使用,推理。 |
| 385 | If you need it to be able to analyze images, you need vision. | 如果需要它才能分析图像,就需要视觉. |
| 386 | If you want it to work in an agentic harness | 如果你想让它在 代理化的绳子上工作 |
| 387 | or an agent mode, you need tool calling. | 或代理模式,您需要工具呼叫。 |
| 388 | Hopefully you get the idea. | 希望你知道这个主意 |
| 389 | There's so many models, | 这么多模特儿 |
| 390 | I can't possibly go through all of them, | 我不可能通过他们所有的, |
| 391 | but you get the idea you can browse through here | 但你有这个想法 你可以浏览这里 |
| 392 | and look at the ones that are going to match | 看看那些会匹配的 |
| 393 | your specific hardware and the sizes that make sense. | 您的特定硬件和大小 合理。 |
| 394 | Now, once you download the model, | 现在,一旦你下载模型, |
| 395 | you can view your models from the model tab right here. | 您可以在这里查看模型标签。 |
| 396 | Now, in order to use these models, | 现在,为了使用这些模型, |
| 397 | you do need to load them. | 你确实需要加载它们。 |
| 398 | So if I select a model like Gemma 4, | 所以如果我选一个像Gemma4的模特儿 |
| 399 | and I just bring open this sidebar here, | 我只是把这个侧柱打开, |
| 400 | it will give me some options for actually loading | 它会给我一些选项 实际加载 |
| 401 | and running my model. | 和运行我的模型。 |
| 402 | So what I may wanna do here is go to the load tab | 所以,我想在这里做的是 去负载标签 |
| 403 | and start changing some of the values. | 并开始改变一些值。 |
| 404 | If you're a beginner, don't change anything | 如果你是初学者,不要改变任何东西 |
| 405 | and just run this directly. | 然后直接运行这个。 |
| 406 | The one thing you can't have a look at | 只有一件事你无法看 |
| 407 | is the context size here. | 这里为上下文大小。 |
| 408 | And keep in mind, the larger you make this context size, | 记住,你做的这个背景尺寸越大, |
| 409 | the more room is gonna be taken up in your computer's memory | 在你的电脑记忆中,会占用更多的房间 |
| 410 | because all of this context needs to actually fit | 因为所有这些背景都需要实际适应 |
| 411 | in the GPU memory, right? | 在GPU的记忆,对不对? |
| 412 | Or the memory that you have for running local models. | 或者你运行本地模型的记忆。 |
| 413 | There's a bunch of other settings you can use, | 还有一堆其他的设置你可以使用, |
| 414 | but in this case, | 但在这种情况下, |
| 415 | what we're gonna do is just load the model. | 我们要做的就是装上模型 |
| 416 | Now, when we load the model, | 现在,当我们加载模型, |
| 417 | same thing it asks us for the settings. | 同样的东西 它要求我们设置。 |
| 418 | We're gonna go ahead and load that. | 我们要继续装货 |
| 419 | It will take a second, | 需要一点时间 |
| 420 | and then we will be able to view that here | 然后我们可以在这里看到 |
| 421 | from this terminal view | 从此终端视图 |
| 422 | and also chat with it directly from the chat window. | 还直接从聊天窗口与它聊天。 |
| 423 | You can load multiple models at once, | 你可以马上装上多个模型, |
| 424 | and you'll be able to see the models that are loaded up here | 你会看到装满的模型 |
| 425 | as well as the size and then to check them. | 还有大小 然后检查一下 |
| 426 | So you can see that I'm currently using 5.58 gigabytes | 所以,你可以看到,我目前使用5.58千兆字节 |
| 427 | of the 63 gigabytes of RAM that I have. | 我拥有63千兆字节的RAM |
| 428 | However, we're talking about GPU memory here, | 但是,我们说的是GPU的记忆, |
| 429 | so that's not really 100% accurate. | 所以这不是真的100%准确。 |
| 430 | So if we go here to the terminal, | 如果我们到终点站 |
| 431 | we can now see that this model is loaded. | 我们现在可以看到这个模型已经装满了。 |
| 432 | I can view all of the API stuff for this | 我可以看所有API的东西 |
| 433 | in case I'm a developer and I wanna directly chat with it | 以防我是开发商 我想直接和它聊天 |
| 434 | using something like a curl command. | 使用类似卷曲命令。 |
| 435 | If that doesn't make sense to you, don't worry. | 如果你觉得不合理 别担心 |
| 436 | If you just wanna chat with the model, | 如果你想和模特儿谈谈 |
| 437 | you can go over to the chat view. | 你可以去聊天视图。 |
| 438 | So from here, we'll press new chat. | 因此,从这里,我们将按新的聊天。 |
| 439 | We're just gonna select the model that's already loaded. | 我们只是要选择 已经上膛的模型。 |
| 440 | 因此谷歌Gemma 4. | |
| 441 | We can modify things here, for example, | 我们可以在这里修改,比如, |
| 442 | like the system prompt if we wanna do that. | 如果我们想这么做 系统就会启动 |
| 443 | And then we can just start chatting directly with the model | 然后我们就可以直接和模型聊天了 |
| 444 | just like we would inside of something like chat GPT. | 就像我们在里面 喜欢聊天GPT。 |
| 445 | Now you can see this one is extremely fast, right? | 现在你可以看到这个非常快,对不对? |
| 446 | We're getting 120 tokens per second | 我们每秒得到120个金币 |
| 447 | because it's very small. | 因为它很小。 |
| 448 | And again, I have high memory bandwidth | 再说一遍,我的记忆带宽很高 |
| 449 | because I'm using a dedicated GPU. | 因为我在使用专用的GPU. |
| 450 | You can load multiple models | 您可以装入多个模型 |
| 451 | as long as they all fit into the memory here. | 只要他们都融入这里的记忆 |
| 452 | And then again, you can adjust all of the parameters. | 然后,你可以调整所有参数。 |
| 453 | And if you want to, you can start using them | 如果你想,你可以开始使用它们 |
| 454 | from this server, which is useful, especially for coding. | 来自此服务器, 特别是用于编码 。 |
| 455 | And you can see the full logs of everything that's gone on, | 你可以看到所有事情的完整记录 |
| 456 | tokens per second speed. | 每秒速度的符号。 |
| 457 | You get the idea. | 你得到这个想法。 |
| 458 | LM Studio is very good. | LM工作室非常不错. |
| 459 | There's a lot you can do with it. | 诸法无量. |
| 460 | And if you want a full tutorial, | 如果你想要一个完整的教程, |
| 461 | leave a comment down below and I will go into it. | 下面留点意见,我再谈 |
| 462 | So the next tool on my list here is Olama. | 我名单上的下一个工具是奥拉玛。 |
| 463 | Now this is a little bit more popular for developers | 这对开发者来说更受欢迎 |
| 464 | 它和LM Studio有很相似之处 | |
| 465 | However, it's a little bit less visual | 然而,它有点少视觉 |
| 466 | and gives you a bit less control. | 让你少点控制力 |
| 467 | Now Olama is a very popular way for downloading | 现在奥拉玛是一个非常流行的下载方式 |
| 468 | and running local models. | 并运行本地模型。 |
| 469 | In order to use it, you do need to download the tool. | 为了使用,您确实需要下载工具. |
| 470 | 所以你可以直接去olama.com | |
| 471 | and it will be available inside of your terminal | 在你的终端里可以找到它 |
| 472 | as a command. | 作为命令。 |
| 473 | So if you type Olama in your terminal once it's downloaded, | 所以,如果你打 奥拉玛在你的终端 一旦下载, |
| 474 | sorry, you should see something like this | 对不起,你应该看看这样的东西 |
| 475 | where you can launch it for all kinds of tools | 在那里你可以推出它 各种工具 |
| 476 | or you can directly chat with different models. | 或者你可以直接和不同的模型聊天。 |
| 477 | Now, as well as the terminal or CLI based tool, | 以及终端或基于CLI的工具 |
| 478 | there is also a visual tool that you can open | 还有一个可以打开的视觉工具 |
| 479 | when you download the desktop application. | 当您下载桌面应用程序时。 |
| 480 | From here again, you can launch Olama | 再来一次,你可以发射奥拉玛 |
| 481 | inside of any of these harnesses | 在任何这些绳子里面 |
| 482 | and use models that you've downloaded. | 并使用你下载过的模型。 |
| 483 | You can go into the settings, right? | 你可以进入环境,对不对? |
| 484 | Or you can actually just start chatting | 或者你可以开始聊天 |
| 485 | with different models by selecting one | 通过选择一个来使用不同的模型 |
| 486 | of the ones that you have. | 你拥有的。 |
| 487 | Now, in order to download models in Olama, | 现在 为了在奥拉玛下载模型 |
| 488 | what you're gonna do is start by finding the model | 你要做的是从找到模型开始 |
| 489 | that you want. | 说你想说的 |
| 490 | So you're gonna have to go to the Olama hub. | 所以你得去奥拉玛中心 |
| 491 | So by doing that, you can go Olama and then Models. | 因此,通过这样做,你可以去奥拉玛和模型。 |
| 492 | From here, there's a bunch of models | 从这里,有一群模特 |
| 493 | that are available for Olama. | 给奥拉玛的 |
| 494 | Same thing, you can search through them. | 同样的事情,你可以搜索 通过他们。 |
| 495 | You can ask ChatGBT to help you find one. | 你可以请ChatGBT帮助你找到一个. |
| 496 | And if you find a model that you want, | 如果你找到一个你想要的模型, |
| 497 | so actually let's go maybe NemoTron 3.5 Lightning here, | 因此,实际上让我们去 也许尼莫特龙3.5闪电在这里, |
| 498 | we can see all the different sizes. | 我们可以看到所有不同的尺寸。 |
| 499 | We now understand what B stands for | 我们现在明白B代表什么 |
| 500 | and what quantization is, right? | 而什么是量化,对不对? |
| 501 | So we can have a look at them here | 好让我们看看这里 |
| 502 | and we can pull them directly inside of Olama. | 我们可以直接把他们拉进奥拉玛 |
| 503 | So the way this works is the following. | 因此,这个方法是: |
| 504 | First, you can type Olama list. | 首先,你可以输入奥拉玛列表. |
| 505 | If you type list, this will show you all the models | 如果您输入列表, 这将显示所有模型 |
| 506 | you currently have downloaded. | 您已下载 。 |
| 507 | And then if you wanna pull a model, you'll type Olama pull | 然后,如果你想拉一个模型, 你会打奥拉玛拉 |
| 508 | and then the model ID that you found from the model hub. | 然后从模型中心找到的模型ID。 |
| 509 | This is gonna download it to your computer | 这个会下载到你的电脑上 |
| 510 | and then allow you to start using it. | 然后让你开始使用它。 |
| 511 | If you wanna run a model, you can type Olama run | 如果你想运行一个模型,你可以打 奥拉玛运行 |
| 512 | and then put the model ID. | 然后把模式ID。 |
| 513 | So I'm gonna put NemoTron 3 like this | 所以我要把尼莫特龙3这样 |
| 514 | and it will start the model. | 它会启动模型。 |
| 515 | It will load it into my computer's memory. | 它会把它装进我电脑的记忆里 |
| 516 | This is why we'll always take a second at the beginning | 这就是为什么我们总是在开始的时候花一秒钟 |
| 517 | cause it actually needs to load it. | 因为它实际上需要加载它。 |
| 518 | Then you can start chatting with it | 那你可以开始跟它聊天了 |
| 519 | directly from this view. | 直接来自此视图。 |
| 520 | So here you can see, I can type something like hello world | 所以这里你可以看到,我可以打一些东西 像你好世界 |
| 521 | and I can just directly start using this local model. | 我可以直接开始使用这个本地模型。 |
| 522 | Now, as well as that, I can do that from this terminal view. | 现在,还有,我可以从这个终端的角度来做。 |
| 523 | So if I go to, let's go NemoTron nano or something, | 所以,如果我去, 让我们去尼莫特龙纳米什么的, |
| 524 | I can type hello. | 我可以打个招呼。 |
| 525 | Same thing, we need to wait for it to be loaded | 同样的事情,我们需要等待它被装上 |
| 526 | and then it will give us a response. | 然后它将给我们一个回应。 |
| 527 | Sometimes it takes a second, especially on the first load | 有时要花点时间 尤其是第一次 |
| 528 | but you can see we get the thinking | 但你可以看到 我们得到的想法 |
| 529 | and then we get the response. | 然后我们得到回应。 |
| 530 | And if we go back here, hello, how can I assist you today? | 如果我们回到这里,你好,我有什么可以帮助你今天? |
| 531 | Now this is great, | 现在,这是伟大的, |
| 532 | but Olama will also expose all of its services on an API. | 但奥拉玛也会在API上曝光它的所有服务. |
| 533 | So for example, if I type Olama help, | 比如,如果我输入奥拉玛的帮助, |
| 534 | you're gonna see an option | 你会看到一个选择 |
| 535 | of all of the different things that you can do. | 所有不同的事情 你可以做的。 |
| 536 | As long as Olama is running, | 只要奥拉玛在跑 |
| 537 | it will actually serve all of its models | 它实际上将服务于它的所有模式 |
| 538 | available on a default port. | 在默认端口上可用。 |
| 539 | I don't remember exactly what the port is | 我不记得港口到底是什么 |
| 540 | but I believe it's something like 11,434, | 但我相信是11,434号 |
| 541 | which means that you're actually able to send curl requests | 也就是说你实际上可以发出卷曲请求 |
| 542 | and use Olama from other tools | 并使用其他工具中的Olama |
| 543 | as long as it's running in the background. | 只要它运行在背景。 |
| 544 | I'm not gonna go into a full tutorial of it | 我不会去一个完整的辅导它 |
| 545 | but if you're a developer and you understand | 但如果你是一个开发者 你明白 |
| 546 | what a REST API is, | 何等为雷斯特API, |
| 547 | Olama provides that already with access | 奥拉玛已经提供 已经进入 |
| 548 | to all of the models. | 给所有的模型。 |
| 549 | It will automatically load any model that you ask it for | 它会自动加载任何您想要的模型 |
| 550 | whenever you try to send a request to it. | 当你试图向它发出一个请求。 |
| 551 | It actually has what's called an OpenAI compatible API | 它实际上有所谓的OpenAI兼容API |
| 552 | which means that you will be able to send requests | 这意味着您将可以发送请求 |
| 553 | in the same format that you would | 格式与您将采用的格式相同 |
| 554 | 像ChatGBT或者Anthropic之类的 | |
| 555 | Anyways, that is Olama. | 总之,那是奥拉玛。 |
| 556 | Let's go to the next example. | 复次举譬. |
| 557 | So the next tool on my list here is the Docker Model Runner. | 我名单上的下一个工具是Docker模型运行器。 |
| 558 | Now this is available as an experimental feature | 现在这个是实验性的 |
| 559 | 在 Docker 桌面内。 | |
| 560 | There is a bunch of restrictions with it. | 有很多限制。 |
| 561 | However, if you are gonna be doing this | 但是,如果你要这样做 |
| 562 | 在 Linux 机器上 | |
| 563 | and you have Nvidia hardware, it works very well. | 你有Nvidia的硬件, 它的工作非常好。 |
| 564 | It can work on CPU as well, I believe, | CPU也有用,我相信, |
| 565 | however, it's extremely slow. | 然而,这是极其缓慢的. |
| 566 | So with Docker Model Runner, | 因此,与多克模型运行者, |
| 567 | if you go into Docker desktop, there is some settings | 如果您进入 Docker 桌面, 有些设置 |
| 568 | I believe you need to enable this experimental feature. | 我相信你需要启用这个实验功能。 |
| 569 | You should see this Models tab. | 你应该看看这个模型标签。 |
| 570 | From the Models tab, you'll be able to go to Docker Hub | 从模型标签,你可以去Docker枢纽 |
| 571 | and then here you can pull all of the same type of models | 然后在这里,你可以拉 所有相同的型号 |
| 572 | as you would be able to inside of like Olama or LM Studio. | 就像奥拉玛或LM工作室 |
| 573 | Once you have a model here, you can chat with it directly. | 一旦你有一个模型在这里,你可以直接与它聊天. |
| 574 | So I have Gemma 3, for example, | 所以我有Gemma 3,例如, |
| 575 | same thing, it will automatically load the model for me. | 同样的,它会自动加载模型给我。 |
| 576 | And then I can type something like hello. | 然后我可以打一个像你好的东西。 |
| 577 | Now, similarly to all of the other tools, | 现在,和所有其他工具一样, |
| 578 | this will also expose a REST API on a different port. | 这也将会在不同的端口暴露一个REST API. |
| 579 | I believe it's 12,434 or 343 | 我想是12,434或343个 |
| 580 | that will allow you the ability to chat with these models | 这样你就能和这些模型聊天 |
| 581 | without being directly inside of this interface. | 而不直接在此界面内。 |
| 582 | You can also inspect the model, | 你也可以检查模型, |
| 583 | see all the information about it, et cetera. | 见一切诸法等. |
| 584 | What's interesting about the Docker Model Runner | Docker模型运行器有什么有趣的 |
| 585 | is that it actually treats models like containers. | 实际上它把模型当成容器 |
| 586 | What that means is that you can write Docker files, | 这意味着你可以写Docker文件, |
| 587 | you can write compose files, | 你可以写作曲文件, |
| 588 | and you can actually have models shipped directly | 也可以直接运出模型 |
| 589 | with your applications and be dependencies exposed | 与您的应用程序并暴露依赖关系 |
| 590 | through Docker kind of services, | 通过多克的服务, |
| 591 | which is a lot more complicated | 这要复杂得多 |
| 592 | than I'm going to get into in this video. | 而不是我要进入这视频。 |
| 593 | But if you do use Docker and you're familiar with this | 但是如果你真的用Docker 你熟悉这个 |
| 594 | and you use it for your apps and you want local models, | 你用它来做你的应用 你需要本地的模型, |
| 595 | this is a really good way to deploy them. | 这是部署他们的好办法 |
| 596 | Now, I'll also show you that there is a CLI based tool. | 现在,我还要告诉你,有一个基于CLI的工具. |
| 597 | So similarly to what we had before, | 跟我们以前一样 |
| 598 | if I type something like Docker model, | 如果我输入的东西 像Docker模型, |
| 599 | you can see that we can configure, | 你可以看到,我们可以配置, |
| 600 | inspect, install the runner, push RM, | 检查,安装跑者,推RM, |
| 601 | view the models, load, unload. | 查看模型,装货,卸货。 |
| 602 | You guys get the idea | 你们知道这个主意 |
| 603 | and you can view models directly inside of here as well. | 也可以直接在这里查看模型 |
| 604 | So this is a really powerful feature. | 这是一个非常强大的特征。 |
| 605 | And if you want a full tutorial on it, | 如果你想要一个完整的辅导, |
| 606 | I have actually done that on my channel. | 我确实在我的频道上做了 |
| 607 | You can see the easiest way to run LM's locally, | 你可以看到最简单的方法 运行本地的LM, |
| 608 | Docker Model Runner tutorial, | Docker 模型运行器教程, |
| 609 | goes through all of the features | 通过所有的特性 |
| 610 | and we'll even show you all of the Docker files | 我们甚至会给你们看所有的Docker文件 |
| 611 | and how to set it up with the automatic deployment. | 和如何设置它与自动部署。 |
| 612 | 好吧,这是Model Runner。 | |
| 613 | Now let's move to the last one, which is full code. | 现在,让我们移动到最后一个, 这是完整的代码。 |
| 614 | Okay, so the last example I have for you | 好,我给你的最后一个例子 |
| 615 | is actually running models using just code. | 正在使用简单的代码运行模型。 |
| 616 | So this means that we're actually gonna bring | 所以这意味着我们真的会带来 |
| 617 | in our own inference engine, in this case, Llama CPP, | 在我们自己的推论引擎中, 在本案中,Llama CPP, |
| 618 | and not rely on something like Olama | 而不是依赖象奥拉玛一样的东西 |
| 619 | 或者Docker Model Runner 或者LM Studio来为我们做这个 | |
| 620 | Now, the big surprise is that Llama CPP is the engine | 现在,最大的惊喜是Llama CPP是引擎 |
| 621 | that pretty much all of the tools that we just looked at | 几乎所有的工具 我们刚刚看 |
| 622 | are already using, | 已经在使用, |
| 623 | but if we want to invoke it directly ourself, | 但如果我们想直接援引它, |
| 624 | we can do that. | 我们可以做到这一点。 |
| 625 | So for example, you'll see we have this Gwen 2.5 model, | 比如说,你会看到我们有Gwen 2.5模型 |
| 626 | which I've downloaded locally on my computer. | 我从电脑上下载的 |
| 627 | Again, this is literally just a file | 再说一遍,这只是一个文件 |
| 628 | that contains a bunch of numbers. | 里面有一堆数字 |
| 629 | Now, if I want to run this, | 现在,如果我想运行这个, |
| 630 | normally I would need a Llama or something like that, | 通常我需要拉玛之类的东西 |
| 631 | but I can actually write code | 但我可以写代码 |
| 632 | that will invoke it directly for me. | 它会直接为我引用它。 |
| 633 | So you'll see that I can just load the model. | 所以你会看到,我可以只是装上模型。 |
| 634 | I can then create a response using this package. | 然后,我可以用这个软件包创建一个响应。 |
| 635 | And if I just run the code here, | 如果我在这里运行密码 |
| 636 | you will see that I get the following. | 你会看到,我得到以下。 |
| 637 | In my heart, I run AI on my desk running free, | 在我的心里,我运行AI在我的桌子上 免费运行, |
| 638 | and it wrote me write a haiku or whatever you call this | 它写我写一个海库 或任何你所谓的这个 |
| 639 | about running AI on your own computer. | 关于运行AI 在自己的计算机。 |
| 640 | Now I can change this prompt to be, | 现在我可以改变这个速度, |
| 641 | hey, who is Tim or something? | 嘿,谁是蒂姆什么的? |
| 642 | I don't know if that's gonna give us anything meaningful, | 我不知道这会不会给我们带来什么意义 |
| 643 | but let's run this. | 但让我们运行这个。 |
| 644 | And Tim is a character from a popular video game, | 蒂姆是一个流行电子游戏的人物 |
| 645 | whatever, right? | 随便吧? |
| 646 | So this is a very small model. | 这是一个很小的模型。 |
| 647 | Of course, it's not gonna give us good responses, | 当然,它不会给我们好的答复, |
| 648 | but you get the idea. | 但你明白这个想法 |
| 649 | We can run it fully locally. | 我们可以完全在当地运行。 |
| 650 | Now, one thing to keep in mind is that as well as doing this, | 现在,有一件事要记住 除了这样做, |
| 651 | we can actually chat with models | 我们可以和模特儿聊天 |
| 652 | that are running on our own computer | 正在我们自己的电脑上运行 |
| 653 | through services like a Llama. | 像Llama这样的服务。 |
| 654 | So like I was mentioning, | 就像我刚才提到的那样 |
| 655 | if a Llama is installed and running, | 如果一个Llama安装和运行, |
| 656 | you can specify the model that's actually available | 您可以指定实际可用的模型 |
| 657 | 在你下载的Llama。 | |
| 658 | And then similarly to before, | 跟以前一样 |
| 659 | I can run something like this. | 我可以运行这样的东西。 |
| 660 | In this case, it's gonna be a bit slower | 这样的话,会慢一点 |
| 661 | because Llama 2 is much larger. | 因为Llama 2是更大的。 |
| 662 | And you'll see that we actually get the response, right? | 你会看到,我们实际上得到了回应,对不对? |
| 663 | And it says, hey, | 它说,嘿, |
| 664 | someone might choose to run an AI model locally, | 有人可能会选择在当地运行一个AI模型, |
| 665 | blah. | |
| 666 | And it's using that a Llama backend service. | 它使用Llama后端服务。 |
| 667 | And if I wanted to, | 如果我想的话, |
| 668 | I could even change this to the LLM API, right? | 我甚至可以把它改成LLM API,对不对? |
| 669 | 或 LM Studio API 或 Docker 模型运行器 API | |
| 670 | and do the exact same thing right from code. | 并做同样的事情 从代码。 |
| 671 | So this is kind of the more manual method, | 所以,这是一种 更手工的方法, |
| 672 | but most developers are gonna end up managing their models | 但大多数开发者最终会管理他们的模型 |
| 673 | through something like a Llama. | 穿过像Llama的东西。 |
| 674 | And then invoking them in code using a method like this. | 然后用这种方法在代码中引用它们。 |
| 675 | Okay, so that wraps up the demos. | 好吧,这样就结束了演示。 |
| 676 | Now let's talk about which method you should actually use. | 现在我们来谈谈你应该使用哪种方法。 |
| 677 | So here's my honest take. | 所以,这是我诚实的拍摄。 |
| 678 | If you just wanna chat with the model | 如果你想和模特儿谈谈 |
| 679 | and you never wanna see a terminal, | 你从不希望看到一个终端, |
| 680 | then you can use something like LM Studio. | 这样你就可以用LM工作室了 |
| 681 | It's genuinely one of the easiest ways to download models | 这是最容易下载模型的方法之一 |
| 682 | and has some of the most amount of features. | 并且拥有一些数量最多的特征. |
| 683 | If you're a developer and you want a model | 如果你是开发商 想要一个模型 |
| 684 | running on your own machine | 在自己的机器上运行 |
| 685 | that your scripts and apps can talk to, | 你的剧本和应用程序可以说话, |
| 686 | definitely use a Llama. | 绝对要用拉玛 |
| 687 | That's what I reach for most days | 大部分时间我都是这样 |
| 688 | and it works really well locally. | 并且它在当地非常有效。 |
| 689 | If you're already living inside of a Docker container | 如果你已经住在多克容器里 |
| 690 | and you want models sitting in that stack right next to you, | 你想让模特坐在你旁边 |
| 691 | 然后使用Docker Model Runner,对吗? | |
| 692 | It's really good in production | 生产真的很好 |
| 693 | if you're actually building and deploying things out. | 如果你真的在建造和部署东西。 |
| 694 | And lastly, if you wanna understand | 最后,如果你想了解的话 |
| 695 | everything that you're doing and run models yourself | 你所做的一切,你自己经营模特 |
| 696 | in probably the most efficient way, | 也许是最有效的方式, |
| 697 | then you can use your own code to do so. | 那样你就可以用你自己的代码了 |
| 698 | Of course, you don't need to use Python. | 当然,你不需要使用Python。 |
| 699 | This is just a quick example. | 这只是一个简单的例子。 |
| 700 | And with that in mind, just remember | 记住这一点,只要记住 |
| 701 | that all of these tools at the end of the day | 最后这些工具 |
| 702 | are using the same building blocks | 使用相同的组件 |
| 703 | that we talked about earlier. | 我们之前说过的 |
| 704 | They have a model, which again, | 他们有一个模型,再次, |
| 705 | is literally just a bunch of numbers in a file | 字面上只是文件里的一堆数字 |
| 706 | and they have an inference engine | 他们有一个推论引擎, |
| 707 | and a bunch of other fancy features on top of it. | 上面还有一堆其他的花样 |
| 708 | If you understand that, you understand local models | 如果你明白这一点,你就明白本地的模型 |
| 709 | and hopefully this video helped get you off the ground | 希望这段视频能帮助你脱身 |
| 710 | and running your first one on your own device. | 然后用自己的设备运行第一个 |
| 711 | Anyways, guys, that's all that I have for you. | 总之,伙计们,我只有这些了 |
| 712 | If you enjoyed, make sure to leave a like, subscribe, | 如果你喜欢, 确保离开类似,订阅, |
| 713 | and I will see you in the next one. | 我会看到你在下一个。 |
| 714 | See you next time. | 下次见 |
| 715 | Bye! | 再见! |
该视频共有字幕 715 条。解锁更多字幕为会员功能，请移动到 价格
P3 · 【本地AI】手把手教你在电脑上跑AI模型 · 零基础指南 p03 无字幕 (P3)
字幕摘录
| 时间 | 英文 | 中文 |
|---|---|---|
| 0:00 | Everybody is talking about running AI locally, | 大家都在谈论本地的人工智能 |
| 0:03 | but almost nobody explains what it is | 但几乎没人解释是什么 |
| 0:05 | or how you actually get it to work. | 或者你如何真正让它工作。 |
| 0:07 | Now you get these videos where people throw around terms | 现在,你得到这些视频 人们扔周围的条件 |
| 0:09 | like weights or quantization, you know, VRAM, GGUF, | 如重量或量化, 你知道,VRAM,GGUF, |
| 0:13 | and it sounds like you need a PhD | 听起来你需要博士学位 |
| 0:15 | and a $10,000 computer just to try this out. | 和一万块的电脑 只是试试这个。 |
| 0:18 | And that's exactly why most people give up | 这就是为什么大多数人放弃 |
| 0:20 | and just keep paying for subscriptions like ChatGPT. | 继续支付像ChatGPT这样的订阅费. |
| 0:23 | So let me give you the honest one sentence version here. | 所以,让我给你一个诚实的句子。 |
展开字幕全文（710 条）
| 序号 | 英文 | 中文 |
|---|---|---|
| 1 | Everybody is talking about running AI locally, | 大家都在谈论本地的人工智能 |
| 2 | but almost nobody explains what it is | 但几乎没人解释是什么 |
| 3 | or how you actually get it to work. | 或者你如何真正让它工作。 |
| 4 | Now you get these videos where people throw around terms | 现在,你得到这些视频 人们扔周围的条件 |
| 5 | like weights or quantization, you know, VRAM, GGUF, | 如重量或量化, 你知道,VRAM,GGUF, |
| 6 | and it sounds like you need a PhD | 听起来你需要博士学位 |
| 7 | and a $10,000 computer just to try this out. | 和一万块的电脑 只是试试这个。 |
| 8 | And that's exactly why most people give up | 这就是为什么大多数人放弃 |
| 9 | and just keep paying for subscriptions like ChatGPT. | 继续支付像ChatGPT这样的订阅费. |
| 10 | So let me give you the honest one sentence version here. | 所以,让我给你一个诚实的句子。 |
| 11 | That is that local AI is simply a model file | 这就是说本地的AI只是一个模型文件 |
| 12 | that's sitting on your computer and a program that runs it. | 坐在你的电脑和运行程序上 |
| 13 | That's it, no cloud, no API keys, | 就这样,没有云,没有API键, |
| 14 | no internet, no subscription. | 没有互联网,没有订阅。 |
| 15 | Everything else is really just a small detail. | 其他一切都只是个小细节 |
| 16 | So in this video, I'm gonna break down | 因此,在这个视频中,我要打破 |
| 17 | what's actually happening under the hood, | 引擎盖下到底发生了什么 |
| 18 | the real building blocks, and none of the fluff. | 真正的建筑构件, 和没有绒毛。 |
| 19 | And then I'm gonna show you how to run a model | 然后我会告诉你如何运行一个模型 |
| 20 | on your own machine in four completely different ways, | 在你自己的机器上 四种完全不同的方式, |
| 21 | from an app that you download | 从您下载的应用程序中 |
| 22 | all the way to pure low-level code | 一直到纯低层代码 |
| 23 | where you're running it yourself. | 在那里你运行它自己。 |
| 24 | Now by the end of this video, | 现在,在这个视频结束时, |
| 25 | you're gonna understand local AI | 你会明白本地的AI |
| 26 | better than most people that are posting about it. | 比大多数贴报的人要好 |
| 27 | So let's dive in. | 所以让我们潜入。 |
| 28 | So first, let's clear up | 先说清楚 |
| 29 | the most common confusion that I see. | 我所看到的最常见的混乱 |
| 30 | Now that's what's actually different | 这才是真正的不同 |
| 31 | between something like ChatGPT and a local model. | 在像ChatGPT之类的东西 和本地模型之间 |
| 32 | Now when you use ChatGPT, Cloud, Gemini, | 当你使用ChatGPT,云,双子座, |
| 33 | really any of these tools, | 任何这些工具, |
| 34 | what's happening is that you type a message, | 发生的事情是 你输入一个消息, |
| 35 | that message leaves your computer, | 留言留下你的电脑, |
| 36 | it travels over the internet to something like a data center | 它在互联网上游历到一个像数据中心的东西 |
| 37 | where a massive computer that you don't own | 你没有的大型电脑 |
| 38 | runs a giant model, and then the answer | 运行一个巨大的模型,然后答案 |
| 39 | gets streamed back to your screen. | 回到屏幕上 |
| 40 | So your computer did basically nothing. | 所以,你的电脑基本上什么都没做。 |
| 41 | It's really just a window into somebody else's machine. | 它真的只是一个窗口 进入别人的机器。 |
| 42 | Now local AI completely flips that. | 现在本地的AI完全推翻了这一点. |
| 43 | The model, so the actual file | 模型,所以实际文件 |
| 44 | that contains all of the intelligence, | 包含所有情报 |
| 45 | gets downloaded onto your computer. | 下载到电脑上 |
| 46 | And when you ask it a question, | 当你问这个问题时, |
| 47 | your own CPU or GPU is the one that's doing the work. | 您的CPU或GPU是 做这项工作。 |
| 48 | So nothing leaves your machine, | 所以,没有什么离开你的机器, |
| 49 | and that gives you three big advantages. | 这给你三个大优势 |
| 50 | Now first, it's private. | 现在,首先,这是私人的。 |
| 51 | That's because your data never goes anywhere, | 那是因为你的数据从不消失 |
| 52 | it stays on your computer. | 它留在你的电脑上。 |
| 53 | Second, it's free, because there's no subscription | 第二,这是免费的,因为没有订阅 |
| 54 | and there's no per-token cost. | 而且没有每桶成本。 |
| 55 | And lastly, it works offline. | 最后,它的工作离线。 |
| 56 | So if you're in a plane or a coffee shop with terrible Wi-Fi, | 所以,如果你在飞机或咖啡店 用可怕的无线, |
| 57 | doesn't matter, you can use these local models. | 无所谓,你可以使用这些本地模型。 |
| 58 | Now I do wanna be honest here about the trade-offs, | 现在,我想诚实地在这里 关于权衡, |
| 59 | which is that the models that you're gonna be able | 也就是你能够做的模型 |
| 60 | to run at home are much smaller than the frontier models | 在家跑步比边疆模式要小得多 |
| 61 | that you'd use from something like Cloud or OpenAI. | 你会用在云或OpenAI之类的东西上 |
| 62 | But they have gone shockingly good | 但是,他们已经走了惊人的好 |
| 63 | over the last couple of years, | 在过去几年里, |
| 64 | and for a huge amount of everyday tasks, | 和大量日常工作, |
| 65 | even coding tasks, they're more than enough. | 甚至编码任务,它们都足够了。 |
| 66 | So with that said, let's have a look | 所以说,让我们来看看 |
| 67 | at the actual pieces here, | 在实际的作品在这里, |
| 68 | so you understand local models much deeper. | 所以你更了解当地的模型 |
| 69 | Now the first part here is the model itself. | 现在第一部分是模型本身。 |
| 70 | And I wanna be really clear about what a model actually is, | 我想说清楚什么是模特儿 |
| 71 | because this is where some people imagine | 因为这就是一些人想象的 |
| 72 | some kind of magic black box. | 某种魔力黑盒。 |
| 73 | Now a model is literally just a file. | 现在一个模型其实只是一个文件。 |
| 74 | It's a really big file, could be hundreds of gigabytes, | 这是一个非常大的文件,可能是数百千兆字节, |
| 75 | that's full of numbers that we call weights. | 我们称之为重量数 |
| 76 | Now these are billions of numbers | 这些是数十亿的数字 |
| 77 | that got baked in when the model was trained. | 当模特被训练时 被烤熟了 |
| 78 | Now that file doesn't think, it doesn't run, | 现在那个文件不思考,它不运行, |
| 79 | it just sits there on your disk | 它只是坐在你的磁盘上 |
| 80 | like any other file that you would have. | 就像其他文件一样 |
| 81 | And companies like Meta, Google, Alibaba, Mistral, whatever, | 还有梅塔、谷歌、阿里巴巴、米斯特拉尔等公司 |
| 82 | they release these files for free. | 他们免费发布这些文件。 |
| 83 | Now those are your open models | 这些是你的公开模型 |
| 84 | and things that you keep hearing about | 和你们所听到的 |
| 85 | 比如Llama或者Gemma或者Quen或者DeepSeq,你懂这个意思. | |
| 86 | You can literally just download them | 你可以直接下载 |
| 87 | because they are literally just a file | 因为从字面上来说,它们只是一个文件 |
| 88 | full of billions of different numbers. | 充满了数十亿个不同的数字。 |
| 89 | Now when you go to look at these models, | 当你去看这些模型时, |
| 90 | you're gonna see names like four billion, | 你会看到40亿的名字, |
| 91 | eight billion, 70 billion. | 八十亿七百亿. |
| 92 | Now that B is billions, | 现在B是亿, |
| 93 | and it stands for the number of parameters. | 它代表参数的数量。 |
| 94 | Now that's effectively just how many numbers | 现在,这是有效的 只是多少数字 |
| 95 | are inside of that file. | 里面的档案。 |
| 96 | And the rule here is pretty simple. | 这里的规则很简单。 |
| 97 | Generally speaking, more parameters means a smarter model, | 一般来说,更多的参数意味着一个更聪明的模型, |
| 98 | but it also means a bigger file | 但它也意味着一个更大的文件 |
| 99 | that needs more memory and compute to run. | 这需要更多的记忆和计算运行。 |
| 100 | So if you look at an eight billion model, | 所以如果你看看80亿的模型 |
| 101 | this is maybe a few gigabytes in storage. | 这也许是几千兆字节的存储。 |
| 102 | Well, a 70 billion model is a file that's so large | 嗯,一个700亿的模型是一个文件 这是那么大 |
| 103 | that most laptops simply cannot even load it, | 大部分笔记本电脑都装不下 |
| 104 | and some may not even have enough storage to download it. | 而有些人甚至可能没有足够的存储器可以下载. |
| 105 | Now that leads directly into the biggest trick | 现在,这直接 进入最大的诡计 |
| 106 | that happens with local AI. | 这发生在本地AI。 |
| 107 | Now this is something called quantization. | 这是所谓的量化。 |
| 108 | Now it sounds scary and really complicated, | 现在听起来很可怕,很复杂 |
| 109 | but it's really the same idea as compressing a photo. | 但它真的相同的想法 压缩一张照片。 |
| 110 | So you can take these billions of numbers | 这样你就可以拿这些亿万的数字 |
| 111 | and you can store them with less precision. | 并且可以更精确地储存它们。 |
| 112 | And that means that this model file | 这意味着这个模型文件 |
| 113 | is gonna get dramatically smaller | 就会变小 |
| 114 | with barely any quality loss at all. | 几乎没有质量损失。 |
| 115 | So a model that would normally need 16 gigabytes of memory | 所以这个模型通常需要16千兆字节的内存 |
| 116 | in its original form might only need five, six, | 原型可能只需要五,六, |
| 117 | or seven gigabytes after quantization. | 或七千兆字节后 |
| 118 | And when you see the term gguf floating around, | 当你看到这个术语漂浮, |
| 119 | that's basically just the standard file format | 这基本上只是标准文件格式 |
| 120 | for these compressed models. | 这些压缩模型。 |
| 121 | Now this is the entire reason | 这才是全部原因 |
| 122 | that normal computers can even run AI models at all, | 普通电脑甚至可以运行AI模型, |
| 123 | or at least some of the bigger ones. | 或至少一些较大的。 |
| 124 | So when you hear about quantization | 所以,当你听到关于量化 |
| 125 | or a quantized model, | 或被量化的模型, |
| 126 | just think of that as a compressed model | 把这当成一个压缩模型 |
| 127 | that's meant to make it smaller so it's easier to run. | 意思是让它变小 所以它更容易运行。 |
| 128 | When you compress these models, | 当你压缩这些模型时, |
| 129 | you keep almost the exact same performance, | 你保持几乎完全相同的表现, |
| 130 | but again, you just reduce the size drastically, | 再说一次,你只是大大缩小了尺寸 |
| 131 | so it's a lot easier to actually run. | 因此,这很容易真正运行。 |
| 132 | Now the next piece is the inference engine. | 现在下一个是推论引擎。 |
| 133 | And this is the part that almost nobody explains. | 这是几乎没人解释的部分。 |
| 134 | Remember that the model is just a file full of numbers, | 记住,模型只是一个文件 充满数字, |
| 135 | and a file can't run itself. | 并且文件不能自己运行。 |
| 136 | So you need a program | 所以你需要一个程序 |
| 137 | that can actually load these numbers into memory | 可以将这些数字装入内存 |
| 138 | and do the actual math to make the model work | 并做实际的数学 使模型工作 |
| 139 | and predict the next token. | 并预言下一个迹象。 |
| 140 | Now that program is called an inference engine. | 现在这个程序叫做推论引擎. |
| 141 | Now the most famous one is called llama.cpp. | 现在最著名的一个叫做lama.cpp. |
| 142 | And here's the secret, | 这是秘密 |
| 143 | almost every tool that I'm gonna show you today, | 几乎每一个工具 我今天要给你看, |
| 144 | so LM Studio, Olamma, Docker Model Runner, | 因此LM工作室,奥拉姆马,多克模型运行器, |
| 145 | they're all basically just a wrapper | 他们基本上都只是一个包装器 |
| 146 | around engines like this. | 绕着这种引擎 |
| 147 | Now the engine does all of the work, | 现在引擎完成了所有的工作, |
| 148 | and the tool just kind of makes it nice to use, | 而工具只是 那种使它使用, |
| 149 | which you're gonna see later on. | 稍后你会看到的。 |
| 150 | Now the last building block here is your hardware, | 最后一块是你们的硬件 |
| 151 | because there's really only one question | 因为只有一个问题 |
| 152 | that decides what local models | 决定何种地方模式 |
| 153 | you're going to be able to run. | 你会跑的 |
| 154 | Now that is, how much memory do you have on your computer, | 你电脑里有多少记忆 |
| 155 | and how fast is that memory? | 记忆有多快? |
| 156 | Now on a PC with a graphics card, | 现在在一台带显卡的PC上 |
| 157 | so a dedicated graphics card, | 所以一个专用的显卡, |
| 158 | that number is gonna be your VRAM. | 这个号码会是你的VRAM。 |
| 159 | So if you have an Nvidia 3090 or 4090 or 5090, | 所以,如果你有一个Nvidia 3090或4090或5090, |
| 160 | or I'm just naming random GPUs, | 或者我只是命名随机GPU, |
| 161 | you're gonna look at the VRAM on that device. | 你看看那个装置上的VRAM |
| 162 | Now if you're running on a modern Mac computer, | 如果你在现代Mac电脑上运行 |
| 163 | 就像M3或M4一样 任何M系列的MacBook | |
| 164 | then you're just gonna be looking at the amount of RAM | 那你就只能看看内存的数量 |
| 165 | that your computer has. | 你的电脑有。 |
| 166 | And that's because Apple shares its memory with the GPU, | 这是因为苹果公司与GPU共享其记忆, |
| 167 | and has something called unified memory. | 并有所谓的统一记忆。 |
| 168 | Now other devices have different specifications, | 现在其他设备有不同的规格, |
| 169 | but generally speaking, | 但总的来说, |
| 170 | if you have a relatively new computer, | 如果你有一个相对新的计算机, |
| 171 | if it has a dedicated graphics card, | 如果它有一个专用的图形卡, |
| 172 | 你看着VRAM。 | |
| 173 | That's the amount of memory you're gonna have | 这就是你的记忆量 |
| 174 | for running local models. | 用于运行本地模型。 |
| 175 | And if you're on a Mac, again, a modern one, | 如果你在Mac上 又是现代的 |
| 176 | you're looking at the amount of unified memory. | 你在看统一记忆的量 |
| 177 | And the rule of thumb here is pretty simple. | 这里的拇指规则很简单。 |
| 178 | The model file needs to be able to fit | 模型文件需要适合 |
| 179 | inside of that memory that you have | 在你的记忆中 |
| 180 | with a little bit of room to spare. | 还有一点空位 |
| 181 | So roughly speaking, | 所以说, |
| 182 | if you have eight gigabytes of VRAM or unified memory, | 如果你有8千兆字节 VRAM 或统一的内存, |
| 183 | you're gonna be able to run | 你会跑的 |
| 184 | three to four billion parameter models, | 三到40亿个参数模型, |
| 185 | even without being quantized. | 即使没有被量化。 |
| 186 | And if you have 16 gigabytes of memory, | 如果你有16千兆字节的记忆 |
| 187 | you can go up to seven or eight billion parameter models. | 可以达到70或80亿个参数模型。 |
| 188 | And then if you go up to 32 gigabytes, | 然后如果你升到32千兆字节 |
| 189 | you start to be able to get into | 你开始可以进入 |
| 190 | the 14 to 30 billion parameter range. | 140-300亿参数范围。 |
| 191 | And this is where things start to feel genuinely smart, | 而这就是事情开始 感觉真正聪明, |
| 192 | especially for local models. | 特别是当地模特儿 |
| 193 | But keep in mind, | 但记住, |
| 194 | you don't need a monster computer to do any of this. | 你不需要怪物电脑来做这些 |
| 195 | Even with something like your phone, | 即使是用你的手机一样的东西 |
| 196 | you can run small models already. | 你可以运行小模型了。 |
| 197 | And one thing that carries over from cloud AI | 还有从云中传来的一件事 |
| 198 | is that the context windows | 是上下文窗口 |
| 199 | to the model short-term working memory | 到模式短期工作记忆 |
| 200 | is going to actually affect the amount of space | 将会影响空间的大小 |
| 201 | that's being taken up. | 这正在处理中。 |
| 202 | So if you have long conversations, bigger documents, et cetera, | 所以,如果你有长话短说, 更大的文件等等, |
| 203 | that's also going to fill up your memory. | 那也会让你的记忆充满 |
| 204 | Now I'm gonna explain this a little bit more in detail | 现在我要详细解释一下 |
| 205 | because this is super important. | 因为这很重要 |
| 206 | But the basic idea is that the one number | 但基本想法是,一个数字 |
| 207 | you need to be aware of is how much memory, | 你需要知道 有多少记忆, |
| 208 | either VRAM or unified memory, is on your computer. | 无论是VRAM还是统一的内存,都在你的电脑上。 |
| 209 | When you look at a model, whether it's quantized or not, | 当你看一个模型, 无论它是否被量化, |
| 210 | it needs to be able to fit comfortably | 它需要能够舒适地装配 |
| 211 | within that memory range. | 在那个记忆范围内 |
| 212 | So if you have 32 gigabytes of VRAM | 所以,如果你有32千兆字节 VRAM |
| 213 | and a model is 25 gigs, that's fine. | 一个模型是25个工作,没关系。 |
| 214 | You'll be able to run it, right? | 你会运行它,对不对? |
| 215 | However, one thing to keep in mind with memory | 不过,有一件事要记住 |
| 216 | is also the speed of the memory. | 也是内存的速度。 |
| 217 | So while you will be able to run much larger models | 所以当你可以运行 更大的模型 |
| 218 | on things like modern Macs that have 128 gigabytes | 在像现代Macs这样的 有128千兆字节的东西上 |
| 219 | of unified memory, the inference speed of those models | 这些模型的推论速度 |
| 220 | is gonna be much slower than a similar model | 比一个类似的模型慢得多 |
| 221 | that's running on something like a dedicated GPU | 运行在像一个专用GPU的东西上 |
| 222 | 或Nvidia RTX GPU. (中文(简体) ). | |
| 223 | The reason for this is the speed of the memory. | 其原因是记忆的速度. |
| 224 | So while the memory will dictate the size of the model | 所以虽然记忆会决定模型的大小 |
| 225 | and the more memory you have, | 越多的记忆, |
| 226 | the smarter models you can run, | 你能够运行的更聪明的模型, |
| 227 | the memory speed matters for the tokens per second | 每秒符号的内存速度很重要 |
| 228 | and the inference speed | 和推论速度 |
| 229 | that you're going to be able to generate. | 你将有能力生成。 |
| 230 | There's a lot of different techniques here | 这里有很多不同的技术 |
| 231 | and things that I could get into, | 和我可以进入的东西, |
| 232 | but generally speaking, | 但总的来说, |
| 233 | if we talk about dedicated graphics cards, | 如果我们谈论专用显卡, |
| 234 | again, typically in the Nvidia family, | 通常在恩维迪亚家族, |
| 235 | these are much faster, sometimes two or three times faster | 速度快得多 有时快两三倍 |
| 236 | at inference speed, but they usually have less capacity. | 在推论速度上, 但他们通常能力较低。 |
| 237 | So for example, I have 24 gigabytes of VRAM in my 4090 | 比如,我4090年有24千兆字节的VRAM |
| 238 | and it's very fast and can generate 200 tokens per second | 它的速度很快,每秒能产生200个令牌 |
| 239 | for some of the models that I run. | 我运行的一些模型。 |
| 240 | However, I can't run models that are 70 billion parameters | 但我不能运行700亿参数的模型 |
| 241 | like I might be able to on my Mac. | 碞钩ии沉 |
| 242 | However, on my Mac, those models are really slow | 然而,在我的Mac上,这些模型真的很慢 |
| 243 | because the memory speed is significantly slower. | 因为内存速度明显较慢. |
| 244 | So you're going to be looking at memory speed | 所以你要看记忆速度 |
| 245 | as well as memory capacity. | 以及记忆能力。 |
| 246 | Those are the two things that are going to dictate | 这些是两件事要决定的 |
| 247 | what you can do with local AI. | 你对本地的AI能做什么? |
| 248 | And there's always going to be a trade-off | 总是有取舍 |
| 249 | in terms of the size of the model | 从模型的大小来看 |
| 250 | and the inference speeds that you're getting. | 和你的推论速度。 |
| 251 | Typically, smaller models are going to be much faster. | 通常,较小的模型会更快。 |
| 252 | And again, the most important thing | 再说一遍,最重要的是 |
| 253 | is that whatever model you run, | 是无论你运行的模型, |
| 254 | it needs to fit in this memory. | 它需要融入这个记忆。 |
| 255 | While it still can run if it's not in there, | 虽然它仍然可以运行,如果它不在那里, |
| 256 | it's going to be so slow that it's practically unusable. | 速度会太慢 以至于无法使用 |
| 257 | Generally speaking, if you're looking at models | 一般来说,如果你在看模特儿 |
| 258 | between 14 and 35 billion parameters, | 140亿至350亿参数之间, |
| 259 | those are going to be a really good sweet spot. | 这将是一个非常好的甜点。 |
| 260 | They're going to give you pretty decent performance | 他们会给你一个很好的表演 |
| 261 | and you're not going to feel | 你不会觉得 |
| 262 | like you're missing out on too much. | 就像你错过了太多。 |
| 263 | If you go up to huge models like 120 billion parameters, | 如果你达到巨大的模型 像1200亿参数, |
| 264 | 250 billion parameters, | 2500亿参数, |
| 265 | you're talking about needing extremely high-end hardware | 你说的是需要非常高端的硬件 |
| 266 | running at slow inference speeds. | 以缓慢推论速度运行。 |
| 267 | And it's very difficult to actually run those at scale | 很难在规模上运行 |
| 268 | on your own machine. | 在自己的机器上。 |
| 269 | Anyways, with that in mind, let's keep going here. | 无论如何,考虑到这一点, 让我们继续在这里。 |
| 270 | I want to tell you about something really interesting. | 我想告诉你一些有趣的事 |
| 271 | Now, the whole reason you'd even bother | 现在,整个原因 你甚至会烦恼 |
| 272 | running models locally comes down to one main thing, | 本地运行的模型 归根结底是一件事, |
| 273 | which I know you all want, which is control. | 我知道你们都想要的 这就是控制 |
| 274 | You pick the model, you own the setup, | 你选模特儿,你拥有设定 |
| 275 | and nobody can change the deal in. | 没有人可以改变交易。 |
| 276 | But here's the problem, right? | 但问题是,对吧? |
| 277 | Almost every tool that you'd actually want to use | 几乎每个工具 你真正想要使用 |
| 278 | locks you into one provider's model. | 把你锁在一个供应商的模型里 |
| 279 | 现在,这正是Mindhub合作, | |
| 280 | who's the sponsor of today's video, is built to fix. | 谁是今天录影带的赞助商 是用来修复的 |
| 281 | Now, it's open source, free to use, | 现在,它是开源的,自由使用, |
| 282 | and it has a real model router. | 它有一个真正的模型路由器。 |
| 283 | 你可以经营Claw、GPT、Chaminade、Jeepseq | |
| 284 | or the same local models that I've been talking about | 或者我一直在谈论的本地模特儿 |
| 285 | right here, all inside of this workspace. | 在这里,所有在这个工作空间内部。 |
| 286 | So the workflow is simple. | 所以工作流程很简单。 |
| 287 | You brief the built-in agent harness, which is Anton, | 你向内置特工介绍一下 也就是安东 |
| 288 | walk away, and come back to finished work. | 走开,回来完成工作。 |
| 289 | Now, I asked it to research the latest coding models | 现在,我要求它研究最新的编码模型 |
| 290 | and build me a comparison dashboard. | 帮我造一个比较仪表板 |
| 291 | And this is what it came back with, | 这就是它回来的原因 |
| 292 | an actual dashboard that I can open and share, | 一个真正的仪表板 我可以打开和分享, |
| 293 | not just a wall of text. | 不只是文字墙。 |
| 294 | And this is the part that connects | 这是连接的部分 |
| 295 | to everything in this video. | 给这段录像里的所有内容 |
| 296 | When a better model drops, | 当一个更好的模型下降, |
| 297 | whether it's local or a cloud model, | 无论是本地还是云型 |
| 298 | I can swap it in and I don't have to change anything. | 我可以把它换进去,我不需要改变任何东西。 |
| 299 | So I have the same workspace, the same work, | 所以我有相同的工作空间,相同的工作, |
| 300 | and since it's fully open source, | 既然它完全开源 |
| 301 | you can clone the repo, | 你可以复制Repo, |
| 302 | spin it up locally in just a few commands, | 转动到局部 仅几个命令, |
| 303 | or just download the dedicated Mac or Windows app. | 或者只下载专用的Mac或Windows应用程序. |
| 304 | The whole thing runs on your own hardware, | 整个事情运行在你的硬件上, |
| 305 | which is basically the end game | 这基本上是结束游戏 |
| 306 | of what we're testing today. | 今天的测试 |
| 307 | Now, they also have a hosted version, | 现在,他们还有一个主机版本, |
| 308 | but honestly, I love the desktop app | 但老实说,我喜欢桌面应用 |
| 309 | as it's very easy to use. | 因为它很容易使用。 |
| 310 | So I'm gonna leave a link to it in the description. | 所以我要在描述中留下一个链接。 |
| 311 | Try it out. | 试试看 |
| 312 | Again, it really goes nicely with these local models. | 再一次,它真的 与这些当地模型。 |
| 313 | And now, let me show you | 现在,让我告诉你 |
| 314 | how we can actually run local models. | 如何实际运行本地模型。 |
| 315 | So there's a lot of ways to run a local model. | 因此当地模式的运行方式很多. |
| 316 | And just like anything in software, | 就像软件里的任何东西一样 |
| 317 | it really comes down to how much control you want. | 这真的取决于你想控制多少 |
| 318 | So I've broken this into four different tiers. | 所以我把它分为四层。 |
| 319 | Now at the top, we've got LM Studio. | 现在最上面的是LM工作室 |
| 320 | This is a regular desktop app. | 这是普通的桌面应用程序 。 |
| 321 | You can click, you can download, you can touch everything, | 你可以点击,你可以下载, 你可以触摸一切, |
| 322 | and you don't really need to go into the terminal. | 而你真的不需要进入终端。 |
| 323 | Now below that, we have Olama. | 下面,我们有奥拉玛。 |
| 324 | This is a really popular option, especially for developers, | 这是一个非常流行的选择, 特别是对开发者来说, |
| 325 | and it's just one command inside of your terminal | 它只是一个命令 在你的终端内部 |
| 326 | where you can talk with models, spin up a local server. | 您可以与模型交谈, 旋转本地服务器。 |
| 327 | You get the idea. | 你得到这个想法。 |
| 328 | 然后我们有了Docker Model Runner。 | |
| 329 | Now this is really good | 现在,这是真的好 |
| 330 | because it treats models like containers, | 因为它对待模型 像容器, |
| 331 | which is perfect if you're actually gonna be deploying | 如果你真的要部署,那就完美了 |
| 332 | these alongside live applications. | 它们与现场应用同时存在。 |
| 333 | And at the bottom of my list here, we have full code. | 在我的名单底部, 我们有完整的代码。 |
| 334 | This is where you're running a model in pure Python, | 这里是你运行一个模型 在纯Python, |
| 335 | and you see every single piece. | 你可以看到每一块。 |
| 336 | Now, no matter which way you wanna run these models here, | 现在,不管你想用哪种方式运行这些模型, |
| 337 | you're gonna be making three decisions. | 你会做出三个决定 |
| 338 | You're gonna pick a model, | 你要选一个模特儿 |
| 339 | you're gonna pick a size and quantization | 你会选择一个大小和量化 |
| 340 | that fits inside of your memory, | 这符合你的记忆, |
| 341 | and you're gonna decide how you wanna talk to it. | 你要决定怎么跟它说话 |
| 342 | So whether that's a chat window or something like code. | 所以,无论这是一个聊天窗口 还是类似代码的东西。 |
| 343 | If you keep that in mind, | 如果你记住这一点, |
| 344 | every single one of these tools is gonna make sense. | 这些工具中的每一个都会有意义 |
| 345 | So let's start at the top. | 故从上起. |
| 346 | And again, I'm gonna go through all of them | 再说一遍,我要翻遍他们所有 |
| 347 | and show you exactly how to run local models. | 告诉你如何运行本地的模型。 |
| 348 | Let's dive in. | 让我们潜入。 |
| 349 | So the first tool on my list here is LM Studio. | 我的第一个工具是LM工作室。 |
| 350 | I'll give you a quick walkthrough, | 我给你一个快速的走过, |
| 351 | but this is one of the best ways to run local models. | 但这是管理本地模型的最佳方法之一 |
| 352 | Now, once you download the tool again, it's completely free. | 现在,一旦你再次下载工具,它完全免费. |
| 353 | You're gonna be brought into a view | 你会被带入风景 |
| 354 | that looks something like this. | 看起来像这样 |
| 355 | From here, you're gonna go into the model view. | 从这里开始,你会进入模型视图。 |
| 356 | It looks like kind of a robot icon on the left-hand side. | 它看起来像一个机器人图标 在左手边。 |
| 357 | And you'll be able to search | 你会可以搜索 |
| 358 | through all of the available models | 通过所有可用的模式 |
| 359 | that you can download directly here. | 您可以在此直接下载。 |
| 360 | Now, what you'll wanna do is search for a model | 现在,你想做的是寻找一个模型 |
| 361 | that matches the kind of relative size | 匹配相对大小的类型 |
| 362 | or amount of memory that you have. | 或你的记忆量 |
| 363 | Again, if we're talking about larger high-end machines, | 再说一遍 如果我们说的是更大的高端机器 |
| 364 | you can typically get away with 27 billion, 35, | 你通常能逃出270亿 35亿 |
| 365 | 30 billion parameters, et cetera. | 三百亿参数等. |
| 366 | If you have eight gigs of RAM or 16 gigs of RAM, | 如果你有8个工作 或16个工作 RAM, |
| 367 | look for ones that are eight billion parameters | 寻找80亿的参数 |
| 368 | or four billion parameters, much smaller sizes. | 或40亿参数,大小小得多。 |
| 369 | So for example, we have Gwen 3.8. | 例如,我们有格温3.8。 |
| 370 | You'll notice that if I click on this, | 你会注意到,如果我点击这个, |
| 371 | I can view different levels of quantization | 我可以看到不同层次的量化 |
| 372 | and see the change in size right here. | 看这里大小的变化 |
| 373 | You also see kind of some icons or indications | 您还看到一些图标或指示 |
| 374 | of which model is the best for you based on your hardware. | 根据你的硬件,哪个型号最适合你。 |
| 375 | So you can see when it says full GPU offload possible | 所以你可以看到,当它说全GPU卸载的可能性 |
| 376 | and a little thumbs up, | 并竖起一小拇指, |
| 377 | that's the one that you would wanna go with. | 那是你想去的那个 |
| 378 | And notice this number of Q4, | 注意这个Q4的号码 |
| 379 | that's the level of precision or the quantization level. | 这是精确度或量化度。 |
| 380 | So the lower, so like Q4, Q2, Q1, | 所以越低,就像Q4,Q2,Q1, |
| 381 | the more quantization you have, right? | 越多的量化,对不对? |
| 382 | So if you have Q8, you can see this is bigger | 所以,如果你有Q8,你可以看到 这是更大的 |
| 383 | than Q6 or Q4, and it's a pretty drastic difference. | 比Q6或Q4, 这是一个相当巨大的差异。 |
| 384 | You're also gonna wanna look at the capabilities, | 你也想看看他们的能力 |
| 385 | so vision, tool use, reasoning. | 因此,视觉,工具的使用,推理。 |
| 386 | If you need it to be able to analyze images, | 如果你需要它来分析图像 |
| 387 | you need vision. | 你需要见识 |
| 388 | If you want it to work in an agentic harness | 如果你想让它在 代理化的绳子上工作 |
| 389 | or an agent mode, you need tool calling. | 或代理模式,您需要工具呼叫。 |
| 390 | Hopefully you get the idea. | 希望你知道这个主意 |
| 391 | There's so many models, | 这么多模特儿 |
| 392 | I can't possibly go through all of them, | 我不可能通过他们所有的, |
| 393 | but you get the idea you can browse through here | 但你有这个想法 你可以浏览这里 |
| 394 | and look at the ones that are going to match | 看看那些会匹配的 |
| 395 | your specific hardware and the sizes that make sense. | 您的特定硬件和大小 合理。 |
| 396 | Now, once you download the model, | 现在,一旦你下载模型, |
| 397 | you can view your models from the model tab right here. | 您可以在这里查看模型标签。 |
| 398 | Now, in order to use these models, | 现在,为了使用这些模型, |
| 399 | you do need to load them. | 你确实需要加载它们。 |
| 400 | So if I select a model like Gemma 4, | 所以如果我选一个像Gemma4的模特儿 |
| 401 | and I just bring open this sidebar here, | 我只是把这个侧柱打开, |
| 402 | it will give me some options for actually loading | 它会给我一些选项 实际加载 |
| 403 | and running my model. | 和运行我的模型。 |
| 404 | So what I may wanna do here is go to the load tab | 所以,我想在这里做的是 去负载标签 |
| 405 | and start changing some of the values. | 并开始改变一些值。 |
| 406 | If you're a beginner, don't change anything | 如果你是初学者,不要改变任何东西 |
| 407 | and just run this directly. | 然后直接运行这个。 |
| 408 | The one thing you can't have a look at | 只有一件事你无法看 |
| 409 | is the context size here. | 这里为上下文大小。 |
| 410 | And keep in mind, the larger you make this context size, | 记住,你做的这个背景尺寸越大, |
| 411 | the more room is gonna be taken up in your computer's memory | 在你的电脑记忆中,会占用更多的房间 |
| 412 | because all of this context needs to actually fit | 因为所有这些背景都需要实际适应 |
| 413 | in the GPU memory, right? | 在GPU的记忆,对不对? |
| 414 | Or the memory that you have for running local models. | 或者你运行本地模型的记忆。 |
| 415 | There's a bunch of other settings you can use, | 还有一堆其他的设置你可以使用, |
| 416 | but in this case, what we're gonna do | 但在这种情况下,我们该怎么办 |
| 417 | is just load the model. | 只是装上模型。 |
| 418 | Now, when we load the model, | 现在,当我们加载模型, |
| 419 | same thing it asks us for the settings. | 同样的东西 它要求我们设置。 |
| 420 | We're gonna go ahead and load that. | 我们要继续装货 |
| 421 | It will take a second, | 需要一点时间 |
| 422 | and then we will be able to view that here | 然后我们可以在这里看到 |
| 423 | from this terminal view and also chat with it | 从此终端视图并与之聊天 |
| 424 | directly from the chat window. | 直接从聊天窗口。 |
| 425 | You can load multiple models at once, | 你可以马上装上多个模型, |
| 426 | and you'll be able to see the models | 你会看到模特儿 |
| 427 | that are loaded up here as well as the size | 这里装满的,还有大小 |
| 428 | and then to check them. | 然后检查他们。 |
| 429 | So you can see that I'm currently using 5.58 gigabytes | 所以,你可以看到,我目前使用5.58千兆字节 |
| 430 | of the 63 gigabytes of RAM that I have. | 我拥有63千兆字节的RAM |
| 431 | However, we're talking about GPU memory here, | 但是,我们说的是GPU的记忆, |
| 432 | so that's not really 100% accurate. | 所以这不是真的100%准确。 |
| 433 | So if we go here to the terminal, | 如果我们到终点站 |
| 434 | we can now see that this model is loaded. | 我们现在可以看到这个模型已经装满了。 |
| 435 | I can view all of the API stuff for this | 我可以看所有API的东西 |
| 436 | in case I'm a developer and I wanna directly chat with it | 以防我是开发商 我想直接和它聊天 |
| 437 | using something like a curl command. | 使用类似卷曲命令。 |
| 438 | If that doesn't make sense to you, don't worry. | 如果你觉得不合理 别担心 |
| 439 | If you just wanna chat with the model, | 如果你想和模特儿谈谈 |
| 440 | you can go over to the chat view. | 你可以去聊天视图。 |
| 441 | So from here, we'll press new chat. | 因此,从这里,我们将按新的聊天。 |
| 442 | We're just gonna select the model that's already loaded. | 我们只是要选择 已经上膛的模型。 |
| 443 | 因此谷歌Gemma 4. | |
| 444 | We can modify things here, for example, | 我们可以在这里修改,比如, |
| 445 | like the system prompt if we wanna do that. | 如果我们想这么做 系统就会启动 |
| 446 | And then we can just start chatting directly with the model | 然后我们就可以直接和模型聊天了 |
| 447 | just like we would inside of something like chat GPT. | 就像我们在里面 喜欢聊天GPT。 |
| 448 | Now you can see this one is extremely fast, right? | 现在你可以看到这个非常快,对不对? |
| 449 | We're getting 120 tokens per second | 我们每秒得到120个金币 |
| 450 | because it's very small. | 因为它很小。 |
| 451 | And again, I have high memory bandwidth | 再说一遍,我的记忆带宽很高 |
| 452 | because I'm using a dedicated GPU. | 因为我在使用专用的GPU. |
| 453 | You can load multiple models | 您可以装入多个模型 |
| 454 | as long as they all fit into the memory here. | 只要他们都融入这里的记忆 |
| 455 | And then again, you can adjust all of the parameters. | 然后,你可以调整所有参数。 |
| 456 | And if you want to, you can start using them | 如果你想,你可以开始使用它们 |
| 457 | from this server, which is useful, especially for coding. | 来自此服务器, 特别是用于编码 。 |
| 458 | And you can see the full logs of everything that's gone on, | 你可以看到所有事情的完整记录 |
| 459 | tokens per second speed. | 每秒速度的符号。 |
| 460 | You get the idea. | 你得到这个想法。 |
| 461 | LM Studio is very good. | LM工作室非常不错. |
| 462 | There's a lot you can do with it. | 诸法无量. |
| 463 | And if you want a full tutorial, | 如果你想要一个完整的教程, |
| 464 | leave a comment down below and I will go into it. | 下面留点意见,我再谈 |
| 465 | So the next tool on my list here is Olama. | 我名单上的下一个工具是奥拉玛。 |
| 466 | Now this is a little bit more popular for developers | 这对开发者来说更受欢迎 |
| 467 | 它和LM Studio有很相似之处 | |
| 468 | However, it's a little bit less visual | 然而,它有点少视觉 |
| 469 | and gives you a bit less control. | 让你少点控制力 |
| 470 | Now Olama is a very popular way for downloading | 现在奥拉玛是一个非常流行的下载方式 |
| 471 | and running local models. | 并运行本地模型。 |
| 472 | In order to use it, you do need to download the tool. | 为了使用,您确实需要下载工具. |
| 473 | 所以你可以直接去olama.com | |
| 474 | and it will be available inside of your terminal | 在你的终端里可以找到它 |
| 475 | as a command. | 作为命令。 |
| 476 | So if you type Olama in your terminal once it's downloaded, | 所以,如果你打 奥拉玛在你的终端 一旦下载, |
| 477 | sorry, you should see something like this | 对不起,你应该看看这样的东西 |
| 478 | where you can launch it for all kinds of tools | 在那里你可以推出它 各种工具 |
| 479 | or you can directly chat with different models. | 或者你可以直接和不同的模型聊天。 |
| 480 | Now, as well as the terminal or CLI based tool, | 以及终端或基于CLI的工具 |
| 481 | there is also a visual tool that you can open | 还有一个可以打开的视觉工具 |
| 482 | when you download the desktop application. | 当您下载桌面应用程序时。 |
| 483 | From here again, you can launch Olama | 再来一次,你可以发射奥拉玛 |
| 484 | inside of any of these harnesses | 在任何这些绳子里面 |
| 485 | and use models that you've downloaded. | 并使用你下载过的模型。 |
| 486 | You can go into the settings, right? | 你可以进入环境,对不对? |
| 487 | Or you can actually just start chatting | 或者你可以开始聊天 |
| 488 | with different models by selecting one | 通过选择一个来使用不同的模型 |
| 489 | of the ones that you have. | 你拥有的。 |
| 490 | Now, in order to download models in Olama, | 现在 为了在奥拉玛下载模型 |
| 491 | what you're gonna do is start by finding the model | 你要做的是从找到模型开始 |
| 492 | that you want. | 说你想说的 |
| 493 | So you're gonna have to go to the Olama hub. | 所以你得去奥拉玛中心 |
| 494 | So by doing that, you can go Olama and then models. | 因此,通过这样做,你可以去奥拉玛然后模型。 |
| 495 | From here, there's a bunch of models | 从这里,有一群模特 |
| 496 | that are available for Olama. | 给奥拉玛的 |
| 497 | Same thing, you can search through them. | 同样的事情,你可以搜索 通过他们。 |
| 498 | You can ask ChatGBT to help you find one. | 你可以请ChatGBT帮助你找到一个. |
| 499 | And if you find a model that you want, | 如果你找到一个你想要的模型, |
| 500 | so actually let's go maybe Nemotron 3.5 Lightning here, | 因此,实际上让我们去 也许尼莫特隆3.5闪电在这里, |
| 501 | we can see all the different sizes. | 我们可以看到所有不同的尺寸。 |
| 502 | We now understand what B stands for | 我们现在明白B代表什么 |
| 503 | and what quantization is, right? | 而什么是量化,对不对? |
| 504 | So we can have a look at them here | 好让我们看看这里 |
| 505 | and we can pull them directly inside of Olama. | 我们可以直接把他们拉进奥拉玛 |
| 506 | So the way this works is the following. | 因此,这个方法是: |
| 507 | First, you can type Olama list. | 首先,你可以输入奥拉玛列表. |
| 508 | If you type list, this will show you all the models | 如果您输入列表, 这将显示所有模型 |
| 509 | you currently have downloaded. | 您已下载 。 |
| 510 | And then if you wanna pull a model, you'll type Olama pull | 然后,如果你想拉一个模型, 你会打奥拉玛拉 |
| 511 | and then the model ID that you found from the model hub. | 然后从模型中心找到的模型ID。 |
| 512 | This is gonna download it to your computer | 这个会下载到你的电脑上 |
| 513 | and then allow you to start using it. | 然后让你开始使用它。 |
| 514 | If you wanna run a model, you can type Olama run | 如果你想运行一个模型,你可以打 奥拉玛运行 |
| 515 | and then put the model ID. | 然后把模式ID。 |
| 516 | So I'm gonna put Nemotron 3 like this | 所以我要把尼莫特龙3这样 |
| 517 | and it will start the model. | 它会启动模型。 |
| 518 | It will load it into my computer's memory. | 它会把它装进我电脑的记忆里 |
| 519 | This is why we'll always take a second at the beginning | 这就是为什么我们总是在开始的时候花一秒钟 |
| 520 | cause it actually needs to load it. | 因为它实际上需要加载它。 |
| 521 | Then you can start chatting with it directly from this view. | 然后你可以直接从这个视图开始与它聊天. |
| 522 | So here you can see, I can type something like hello world | 所以这里你可以看到,我可以打一些东西 像你好世界 |
| 523 | and I can just directly start using this local model. | 我可以直接开始使用这个本地模型。 |
| 524 | Now, as well as that, I can do that from this terminal view. | 现在,还有,我可以从这个终端的角度来做。 |
| 525 | So if I go to, let's go Nemotron Nano or something, | 所以,如果我去, 让我们去尼莫特隆纳诺什么的, |
| 526 | I can type hello. | 我可以打个招呼。 |
| 527 | Same thing, we need to wait for it to be loaded | 同样的事情,我们需要等待它被装上 |
| 528 | and then it will give us a response. | 然后它将给我们一个回应。 |
| 529 | Sometimes it takes a second, especially on the first load | 有时要花点时间 尤其是第一次 |
| 530 | but you can see we get the thinking | 但你可以看到 我们得到的想法 |
| 531 | and then we get the response. | 然后我们得到回应。 |
| 532 | And if we go back here, hello, how can I assist you today? | 如果我们回到这里,你好,我有什么可以帮助你今天? |
| 533 | Now this is great, but Olama will also expose | 现在这是伟大的,但奥拉玛也会暴露 |
| 534 | all of its services on an API. | 所有服务在API。 |
| 535 | So for example, if I type Olama help, | 比如,如果我输入奥拉玛的帮助, |
| 536 | you're gonna see an option | 你会看到一个选择 |
| 537 | of all of the different things that you can do. | 所有不同的事情 你可以做的。 |
| 538 | As long as Olama is running, | 只要奥拉玛在跑 |
| 539 | it will actually serve all of its models | 它实际上将服务于它的所有模式 |
| 540 | available on a default port. | 在默认端口上可用。 |
| 541 | I don't remember exactly what the port is | 我不记得港口到底是什么 |
| 542 | but I believe it's something like 11,434 | 但我相信是11,434号 |
| 543 | which means that you're actually able to send | 也就是说你实际上可以发送 |
| 544 | curl requests and use Olama from other tools | 卷曲请求并使用其他工具的 Olama |
| 545 | as long as it's running in the background. | 只要它运行在背景。 |
| 546 | I'm not gonna go into a full tutorial of it | 我不会去一个完整的辅导它 |
| 547 | but if you're a developer and you understand | 但如果你是一个开发者 你明白 |
| 548 | what a REST API is, Olama provides that already | 欧拉玛已经提供了 |
| 549 | with access to all of the models | 能够访问所有模型 |
| 550 | and will automatically load any model that you ask it for | 并自动加载您想要的任何模型 |
| 551 | whenever you try to send a request to it. | 当你试图向它发出一个请求。 |
| 552 | It actually has what's called an open AI compatible API | 它实际上有一个所谓的开放的 AI兼容API |
| 553 | which means that you will be able to send requests | 这意味着您将可以发送请求 |
| 554 | in the same format that you would to something | 和你想的一样 |
| 555 | 像ChatGBT或Anthropic一样。 | |
| 556 | Anyways, that is Olama, let's go to the next example. | 总之,这是奥拉玛, 让我们来看看下一个例子。 |
| 557 | So the next tool on my list here is the Docker model runner. | 我名单上的下一个工具是Docker模型跑者。 |
| 558 | Now this is available as an experimental feature | 现在这个是实验性的 |
| 559 | 在 Docker 桌面内。 | |
| 560 | There is a bunch of restrictions with it. | 有很多限制。 |
| 561 | However, if you are gonna be doing this | 但是,如果你要这样做 |
| 562 | 在 Linux 机器上 | |
| 563 | and you have Nvidia hardware, it works very well. | 你有Nvidia的硬件, 它的工作非常好。 |
| 564 | It can work on CPU as well I believe | CPU也能用,我相信 |
| 565 | however it's extremely slow. | 但它非常缓慢。 |
| 566 | So with Docker model runner, if you go into Docker desktop | 因此,与Docker模型跑车, 如果你进入Docker桌面 |
| 567 | there is some settings I believe you need to enable | 有一些设置 我想你需要启用 |
| 568 | this experimental feature, you should see this models tab. | 此实验功能, 您应该看到此模型标签 。 |
| 569 | From the models tab, you'll be able to go to Docker hub | 从模型标签,你可以去Docker枢纽 |
| 570 | and then here you can pull all of the same type of models | 然后在这里,你可以拉 所有相同的型号 |
| 571 | as you would be able to inside of like Olama or LM studio. | 就像奥拉玛或LM工作室 |
| 572 | Once you have a model here, you can chat with it directly. | 一旦你有一个模型在这里,你可以直接与它聊天. |
| 573 | So I have Gemma three, for example, same thing | 所以我有Gemma 3,比如说,同样的事情 |
| 574 | it will automatically load the model for me | 它会自动为我装上模型 |
| 575 | and then I can type something like hello. | 然后我可以打出像你好的东西。 |
| 576 | Now, similarly to all of the other tools | 现在,和所有其他工具一样 |
| 577 | this will also expose a rest API on a different port. | 这也将暴露一个休息的API在另一个端口。 |
| 578 | I believe it's 12,434 or 343 that will allow you | 我相信是12,434或343允许你 |
| 579 | the ability to chat with these models | 与这些模型交谈的能力 |
| 580 | without being directly inside of this interface. | 而不直接在此界面内。 |
| 581 | You can also inspect the model, | 你也可以检查模型, |
| 582 | see all the information about it, et cetera. | 见一切诸法等. |
| 583 | What's interesting about the Docker model runner | 多克模特跑车有什么好玩的 |
| 584 | is that it actually treats models like containers. | 实际上它把模型当成容器 |
| 585 | What that means is that you can write Docker files, | 这意味着你可以写Docker文件, |
| 586 | you can write compose files | 您可以写作文件 |
| 587 | and you can actually have models shipped directly | 也可以直接运出模型 |
| 588 | with your applications and be dependencies exposed | 与您的应用程序并暴露依赖关系 |
| 589 | through Docker kind of services, | 通过多克的服务, |
| 590 | which is a lot more complicated | 这要复杂得多 |
| 591 | than I'm going to get into in this video. | 而不是我要进入这视频。 |
| 592 | But if you do use Docker and you're familiar with this | 但是如果你真的用Docker 你熟悉这个 |
| 593 | and you use it for your apps and you want local models, | 你用它来做你的应用 你需要本地的模型, |
| 594 | this is a really good way to deploy them. | 这是部署他们的好办法 |
| 595 | Now I'll also show you that there is a CLI based tool. | 现在,我也会告诉你,有一个基于CLI的工具。 |
| 596 | So similarly to what we had before, | 跟我们以前一样 |
| 597 | if I type something like Docker model, | 如果我输入的东西 像Docker模型, |
| 598 | you can see that we can configure, | 你可以看到,我们可以配置, |
| 599 | inspect, install the runner, push RM, | 检查,安装跑者,推RM, |
| 600 | view the models, load, unload. | 查看模型,装货,卸货。 |
| 601 | You guys get the idea | 你们知道这个主意 |
| 602 | and you can view models directly inside of here as well. | 也可以直接在这里查看模型 |
| 603 | So this is a really powerful feature. | 这是一个非常强大的特征。 |
| 604 | And if you want a full tutorial on it, | 如果你想要一个完整的辅导, |
| 605 | I have actually done that on my channel. | 我确实在我的频道上做了 |
| 606 | You can see the easiest way to run LMS locally, | 你可以看到在本地运行LMS最简单的方式, |
| 607 | Docker模特跑车辅导, | |
| 608 | goes through all of the features | 通过所有的特性 |
| 609 | and we'll even show you all of the Docker files | 我们甚至会给你们看所有的Docker文件 |
| 610 | and how to set it up with the automatic deployment. | 和如何设置它与自动部署。 |
| 611 | Okay, that is model runner. | 好吧,这是模型跑者。 |
| 612 | Now let's move to the last one, which is full code. | 现在,让我们移动到最后一个, 这是完整的代码。 |
| 613 | Okay, so the last example I have for you | 好,我给你的最后一个例子 |
| 614 | is actually running models using just code. | 正在使用简单的代码运行模型。 |
| 615 | So this means that we're actually gonna bring | 所以这意味着我们真的会带来 |
| 616 | in our own inference engine, in this case, llama CPP, | 我们自己的推论引擎, 在这种情况下,拉玛CPP, |
| 617 | and not rely on something like olama | 而不是依赖像奥拉玛一样的东西 |
| 618 | or Docker model runner or LM studio to do this for us. | 或Docker模特跑车 或LM工作室 这样做为我们。 |
| 619 | Now the big surprise is that llama CPP is the engine | 现在最大的惊喜是拉玛CPP是引擎 |
| 620 | that pretty much all of the tools that we just looked at | 几乎所有的工具 我们刚刚看 |
| 621 | are already using, | 已经在使用, |
| 622 | but if we want to invoke it directly ourself, | 但如果我们想直接援引它, |
| 623 | we can do that. | 我们可以做到这一点。 |
| 624 | So for example, you'll see we have this Gwen 2.5 model, | 比如说,你会看到我们有Gwen 2.5模型 |
| 625 | which I've downloaded locally on my computer. | 我从电脑上下载的 |
| 626 | Again, this is literally just a file | 再说一遍,这只是一个文件 |
| 627 | that contains a bunch of numbers. | 里面有一堆数字 |
| 628 | Now, if I want to run this, | 现在,如果我想运行这个, |
| 629 | normally I would need olama or something like that, | 通常我需要欧拉玛之类的东西 |
| 630 | but I can actually write code | 但我可以写代码 |
| 631 | that will invoke it directly for me. | 它会直接为我引用它。 |
| 632 | So you'll see that I can just load the model. | 所以你会看到,我可以只是装上模型。 |
| 633 | I can then create a response using this package. | 然后,我可以用这个软件包创建一个响应。 |
| 634 | And if I just run the code here, | 如果我在这里运行密码 |
| 635 | you will see that I get the following. | 你会看到,我得到以下。 |
| 636 | In my heart, I run AI on my desk running free | 在我的心里,我运行AI在我的桌子上 免费运行 |
| 637 | and it wrote me write a haiku or whatever you call this | 它写我写一个海库 或任何你所谓的这个 |
| 638 | about running AI on your own computer. | 关于运行AI 在自己的计算机。 |
| 639 | Now I can change this prompt to be, | 现在我可以改变这个速度, |
| 640 | hey, who is Tim or something? | 嘿,谁是蒂姆什么的? |
| 641 | I don't know if that's gonna give us anything meaningful, | 我不知道这会不会给我们带来什么意义 |
| 642 | but let's run this. | 但让我们运行这个。 |
| 643 | And Tim is a character from a popular video game, | 蒂姆是一个流行电子游戏的人物 |
| 644 | whatever, right? | 随便吧? |
| 645 | So this is a very small model. | 这是一个很小的模型。 |
| 646 | Of course, it's not gonna give us good responses, | 当然,它不会给我们好的答复, |
| 647 | but you get the idea. | 但你明白这个想法 |
| 648 | We can run it fully locally. | 我们可以完全在当地运行。 |
| 649 | Now, one thing to keep in mind is that as well as doing this, | 现在,有一件事要记住 除了这样做, |
| 650 | we can actually chat with models | 我们可以和模特儿聊天 |
| 651 | that are running on our own computer | 正在我们自己的电脑上运行 |
| 652 | 透过olama等服务. | |
| 653 | So like I was mentioning, | 就像我刚才提到的那样 |
| 654 | if olama is installed and running, | 如果olama安装和运行, |
| 655 | you can specify the model that's actually available | 您可以指定实际可用的模型 |
| 656 | 在你下载的olama。 | |
| 657 | And then similarly to before, | 跟以前一样 |
| 658 | I can run something like this. | 我可以运行这样的东西。 |
| 659 | In this case, it's gonna be a bit slower | 这样的话,会慢一点 |
| 660 | cause llama two is much larger. | 因为拉玛2更大 |
| 661 | And you'll see that we actually get the response, right? | 你会看到,我们实际上得到了回应,对不对? |
| 662 | And it says, hey, someone might choose | 它说,嘿,有人可能会选择 |
| 663 | to run an AI model locally, blah, blah, blah, blah. | 在当地运行一个AI模型,等等,等等,等等。 |
| 664 | And it's using that olama backend service. | 它正在使用 olama后端服务。 |
| 665 | And if I wanted to, I could even change this | 如果我想,我甚至可以改变这个 |
| 666 | 到LLM API,对不对? | |
| 667 | 或 LM 工作室 API 或 Docker 模型跑者 API | |
| 668 | and do the exact same thing right from code. | 并做同样的事情 从代码。 |
| 669 | So this is kind of the more manual method, | 所以,这是一种 更手工的方法, |
| 670 | but most developers are gonna end up managing their models | 但大多数开发者最终会管理他们的模型 |
| 671 | through something like olama. | 穿过像olama的东西。 |
| 672 | And then invoking them in code using a method like this. | 然后用这种方法在代码中引用它们。 |
| 673 | Okay, so that wraps up the demos. | 好吧,这样就结束了演示。 |
| 674 | Now let's talk about which method you should actually use. | 现在我们来谈谈你应该使用哪种方法。 |
| 675 | So here's my honest take. | 所以,这是我诚实的拍摄。 |
| 676 | If you just want to chat with the model | 如果你只想跟模特儿聊天 |
| 677 | and you never want to see a terminal, | 你永远不想看到一个终端, |
| 678 | then you can use something like LM studio. | 那样你就可以用LM工作室 |
| 679 | It's genuinely one of the easiest ways to download models | 这是最容易下载模型的方法之一 |
| 680 | and has some of the most amount of features. | 并且拥有一些数量最多的特征. |
| 681 | If you're a developer and you want a model running | 如果你是开发商,想要一个模型运行 |
| 682 | on your own machine that your scripts and apps can talk to, | 在你自己的机器上 你的脚本和应用程序可以说话, |
| 683 | 绝对使用olama。 | |
| 684 | That's what I reach for most days | 大部分时间我都是这样 |
| 685 | and it works really well locally. | 并且它在当地非常有效。 |
| 686 | If you're already living inside of a Docker container | 如果你已经住在多克容器里 |
| 687 | and you want models sitting in that stack right next to you, | 你想让模特坐在你旁边 |
| 688 | then use Docker model runner, right? | 然后用Docker模型跑车 对吧? |
| 689 | It's really good in production. | 生产上真的很好 |
| 690 | If you're actually building and deploying things out. | 如果你真的在建造和部署东西 |
| 691 | And lastly, if you want to understand everything | 最后,如果你想知道一切的话 |
| 692 | that you're doing and run models yourself | 你做和运行 模特自己 |
| 693 | in probably the most efficient way, | 也许是最有效的方式, |
| 694 | then you can use your own code to do so. | 那样你就可以用你自己的代码了 |
| 695 | Of course, you don't need to use Python. | 当然,你不需要使用Python。 |
| 696 | This is just a quick example. | 这只是一个简单的例子。 |
| 697 | And with that in mind, just remember that all of these tools | 记住这些工具 |
| 698 | at the end of the day are using the same building blocks | 最终是用同样的构件 |
| 699 | that we talked about earlier. | 我们之前说过的 |
| 700 | They have a model, which again, | 他们有一个模型,再次, |
| 701 | is literally just a bunch of numbers in a file | 字面上只是文件里的一堆数字 |
| 702 | and they have an inference engine | 他们有一个推论引擎, |
| 703 | and a bunch of other fancy features on top of it. | 上面还有一堆其他的花样 |
| 704 | If you understand that, you understand local models | 如果你明白这一点,你就明白本地的模型 |
| 705 | and hopefully this video helped get you off the ground | 希望这段视频能帮助你脱身 |
| 706 | and running your first one on your own device. | 然后用自己的设备运行第一个 |
| 707 | Anyways, guys, that's all that I have for you. | 总之,伙计们,我只有这些了 |
| 708 | If you enjoyed, make sure you like, subscribe | 如果你喜欢,确保你喜欢,订阅 |
| 709 | and I will see you in the next one. | 我会看到你在下一个。 |
| 710 | It's Friday. | 礼拜五 |
该视频共有字幕 710 条。解锁更多字幕为会员功能，请移动到 价格