← AI Agents & Vibe Coding
Episode 11··30:30

Your own AI model instead of a subscription: too early for agents, already cheaper elsewhere

A subscription to a strong model costs money, while free models come out almost every week, and by their own tables they are barely worse. It is tempting to download one, run it yourself and stop paying. For an agent's main work this is not the answer yet, and speed is not the reason. A big model simply has nowhere to live at home. A small one makes mistakes slightly more often at each step, and on a long task those mistakes add up to failure. You will understand why you should not trust the tables in announcements, and you will learn how to ask your agent to compare models on thirty to a hundred of your own real examples. And if you do run a model on a laptop, you will learn which setting to check first: by default the model remembers too little text for an agent to work with it. The second half is about where a small model of your own already pays off: on one narrow, repetitive task, if you train it on your own data. Your agent takes care of all the technical work. You will learn how many examples you need and what the hardware costs, and about three traps: the rules of Anthropic and OpenAI on training with their answers, other people's personal data, and the security of someone else's model on your server. At the end, my forecast for when models on your own computer will catch up with the best paid ones.

Transcript

Tap a paragraph to listen from there.

Good evening, startup founders and vibe coders! Today I am going to tell you whether you can run your own neural network on your own computer and drop a two-hundred-dollar monthly subscription. Let me say it right away: for an agent's main work, not yet, and I will explain why speed has nothing to do with it. Then I will show you where a small model of your own really does win, and why in many of my own projects I already manage without the big ones.

This question is coming up everywhere right now. In the last few weeks several open models came out at once, each with a beautiful table in its announcement, and one of them, by its own measurements, nearly catches up with Opus. So many people have a simple thought: why pay if you can download it for free? You will take three things away from this episode. You will understand where the gap between your own models and frontier models comes from. You will learn how to have your agent test any model on your own tasks. And you will know the traps waiting for anyone who decides to train a model of their own.

Let us start with what a model of your own actually is. Some companies publish a model's weights openly. The weights are essentially the trained neural network itself, a huge file full of numbers. You download it, run it on your computer through a program like Ollama, and the model answers you with no internet and no subscription. The best-known family of these models is Qwen from the Chinese company Alibaba. There is Llama from Meta, there is Kimi, there is GLM, and dozens of others.

Open models also have a dark side that many people have heard about. These are the so-called uncensored versions: someone edits the model's weights so that it stops refusing. There are thousands of such builds on the Hugging Face platform, including ones for Qwen. The platform removes some of them, but copies from other authors remain. People say that some use these models to work through certain questions and then bring the result to frontier models. I found no descriptions of such a scheme in open sources, and I will not encourage it: the rules of Anthropic and OpenAI directly forbid getting around their restrictions. Some people do run these builds, but there is also the ordinary Qwen, and from here on I am talking only about it and other ordinary open models.

Now to the main point. An open model means the weights can be downloaded. But that does not mean at all that you can run it at home. Take Kimi K3 from the Chinese company Moonshot and its latest version, K3.1 Pro. According to Moonshot's own tests, it is nearly level with Opus 5.5. Yet its official files weigh more than one and a half thousand gigabytes, and that is already in compressed form.

Why is this a problem? While it works, the model has to sit entirely in the fast memory of the graphics card. Not on the disk, but in that memory, or it will answer painfully slowly. A model can be compressed, a bit like compressing a photo: the file gets smaller, but the fine details are lost. Well, Kimi's files are already compressed, and it still needs more than one and a half terabytes of video memory. No home graphics card and no single Mac can hold that much on its own.

At this point people usually object: wait, in Kimi only a small part of the model works on each word. That is true. It is built like a team of many narrow specialists, and at each step only a few of them switch on. But you have to keep the whole team in memory at once, because you never know in advance who will be needed for the next word.

Now in money. The most modest server that Kimi fits on has eight of the newest server graphics cards. Such a server was offered for about half a million dollars. And for the model to work with the full length of text it was designed for, NVIDIA offers a bigger server, and on previous-generation cards that comes to about one and a half million. So half a million, or even a million dollars, is the honest price of running it properly.

Some enthusiasts build a cluster for this out of four Mac Studio computers with maximum memory. Together that is almost ninety thousand dollars. The model runs on it, but slowly. With an agent like Claude Code, the conversation with the model easily reaches a hundred thousand tokens. According to a measurement from early September, such a cluster needs about seven minutes just to read that much. And that happens at every step of the agent. It is a toy for one patient person, not a working tool.

And here is a funny detail. Through Moonshot's own cloud, the same model costs two dollars per million input tokens and ten per million output tokens. That is cheaper than Opus. So open weights give you nothing at home, and in the cloud this is simply one more paid service.

All right, a big model cannot be run at home. But there are smaller models that fit on a MacBook or on a single gaming graphics card. Why not use them at least for simple tasks? My answer is this: in practice, there is not much choice right now. Even a simple task, if it has to be done without mistakes, today needs a frontier model, even if it is a weaker one. Some inexpensive Sonnet, or OpenAI's small models like GPT Terra or GPT Luna. Neither Qwen nor Llama on a laptop can handle it. And small frontier models are quickly becoming more reliable. About GPT Terra, which came out at the end of October, OpenAI states that its share of made-up answers fell from eighty-eight percent to thirty-eight. Those are the company's own words, but the direction is telling.

The most interesting part is understanding why. Many people think the whole difference is speed. Say my model on a graphics card answers in thirty seconds, fine, I will wait. But even when it answers quickly, it is nowhere near Opus. There are several reasons, and the most important one is how errors add up step by step.

Look at how an agent works. It reads a file, figures out what is in it, makes an edit, runs it, reads the error, edits again. It is a chain of steps, and each step can go wrong. Suppose a model gets each step right ninety-five times out of a hundred. Sounds decent. But it will get through a twenty-step task without a single slip only about a third of the time. And if the model is right at each step ninety-nine times out of a hundred, twenty steps go through eighty-two times out of a hundred.

Think about it: a difference of four points on a single step turns into a more than twofold difference on the whole task. Real life is a little gentler, but the point holds. Researchers noticed one more thing. Models work worse when the context contains their errors from prior turns. One mistake pulls the next one after it.

That is why speed has nothing to do with it. A fast model simply reaches failure faster. If it is slightly weaker at each step, it fails a long task almost every time. And an agent's work on your project is exactly that: long tasks.

The second reason is the amount of knowledge. Researchers calculated that a model can store about two bits of knowledge per parameter, that is, per number inside it. Fewer numbers, less knowledge. And when a model does not know something, it does not stay silent, it makes things up. OpenAI itself writes about this in the description of its open models: smaller models have less world knowledge than larger frontier models and tend to hallucinate more. On its test of factual questions, the smaller model, the one that fits on a single gaming graphics card, made up an answer ninety-one times out of a hundred. The larger model did so seventy-eight times. Still a lot, but the difference is noticeable.

The third reason is that same compression. To make a model fit in a laptop's memory, its numbers are made coarser. Light compression is almost unnoticeable, with losses under one percent. Heavy compression already hurts: on long-text tasks, in one study, the losses reached fifty-nine percent. And here is what matters for you and me: in languages other than English, accuracy can drop up to five times more than in English. If you talk to the model in Russian, compression hits you harder.

And with the coarsest compression, the model starts going in circles: long reasoning full of repetition, thoughts that break off, and still no answer. Look at what is being published these days as a big model for a single graphics card. For example, a cut-down version of GLM: a quarter of its specialists were removed, and it was heavily compressed on top of that. The authors themselves write: "Two losses stack." And they give no comparison with the full version.

The fourth reason is working with tools. An agent is constantly calling something: read a file, run a command, search the internet. In one such test, the small Qwen succeeded on average about sixty-two times out of a hundred. But on tasks where calls come one after another, as in an agent's real work, only thirty-five times. A small model handles one step reasonably well, but not a chain.

The fifth reason is very down to earth, and I think it is the most common one. It is the memory for text, called the context. That is how much text the model keeps in its head at once. On paper, a model may have two hundred fifty-six thousand tokens, or even a million. But by default, Ollama adjusts this size to the graphics card. If you have less than twenty-four gigabytes of video memory, it sets just four thousand tokens. Meanwhile Ollama itself writes that agents and coding tools should be set to at least sixty-four thousand tokens.

So on an ordinary laptop, by default, the model remembers sixteen times less than an agent needs. The agent forgets the beginning of its own work and starts going in circles. People see this and say: the local model is dumb. But it simply cannot see half of the task.

So if you do decide to try a local model, remove this trap first. Otherwise you will be judging not the model, but its cut-down memory. Tell your agent: check what context size is set for the model in Ollama, set it to at least sixty-four thousand if there is enough memory, and calculate how much memory that will take. The context itself takes up space, and quite a lot of it. And add: pick a lightly compressed version of the model that fits in my memory together with that context.

But there is also a sixth reason, the deepest one. A model does not learn to work as an agent from books. It is trained on a huge number of tasks where the result can be checked, and it is rewarded for a correct solution. People build these tasks by hand, and that is very expensive. According to researchers' estimates, one such task costs from two hundred to two thousand dollars. A copy of a complex working service for training costs about three hundred thousand. And journalists wrote that Anthropic had discussed spending more than a billion dollars on such training environments.

This is the main wealth of the frontier labs, and you cannot download it along with the weights. Open models catch up partly by copying. In February, Anthropic reported about twenty-four thousand fake accounts used to pump out Claude's answers, and they were aimed precisely at agent work and code. But copying answers does not fully transfer the skill. And into a small model it transfers even worse.

Here you might say: but I have seen tables where an open model is almost like Opus. Let us sort out the tables. That same Kimi, by Moonshot's own measurements, scores sixty-five point seven against sixty-six point four for Opus 5.5. Almost a tie. But on an independent math test, Kimi scores fifty-four percent, while frontier models score from eighty to eighty-five.

It turned out even more clearly with one well-known independent ranking. At the end of October, its authors made more than half of the tasks private: they cannot be peeked at in advance, and a model cannot be tuned to them. And the gap between the best open and closed models immediately grew from three points to twelve. Four times larger, as soon as the tasks stopped being known in advance. The new American model Ripple shows eighty point five points on an agent test in its announcement, while an independent summary gives it about fifty-seven. They count a little differently there, but the direction is clear. To the authors' credit, they write themselves that replicating benchmark scores has been challenging.

And there is one observation that, frankly, surprised me. One researcher ran the same open models in different agent harnesses. A harness is the program around the model that gives it tools and guides it through the task, for example Claude Code or Codex. The result of one and the same model changed by tens of points from one harness to another. And different models in the same harness also gave very different results. The conclusion is simple: if you plug an open model into Claude Code, you will not get Claude's level. The harness is good, but the ability to solve tasks sits in the model itself.

Now, honestly, about the other side. Frontier models are far from all-powerful too. In the last two weeks, three independent tests on real work came out at once. The first took two hundred forty real jobs from a freelance marketplace, and the results were judged by human experts. The best turned out to be the new Fable 5.2: it completed forty-one percent of the jobs. Opus 5.5 completed twenty-one. A year ago the best result was two and a half percent. The progress is huge, but more than half of the work still did not get done.

In the second test, agents improved real web applications based on user requests. The best model met about seventy percent of the requirements on average. But it got things fully right only twenty-six times out of a hundred. And on the old test, the same agents scored over ninety-nine percent. In the third, the hardest test, on professional work, not a single agent got above twelve percent.

What does this mean for you? If even the best models fail most real work, a small model will do worse. And even more important: you cannot trust the table from an announcement. You can only trust a test on your own tasks.

Here is the advice I consider the most important in this episode. Before you switch models on any task, tell your agent: collect thirty to a hundred real examples of this task from my data with correct answers, run them through the current model and through the new one, count how many are solved fully correctly, and show me every failure. Keep the grading simple: pass or fail, no scales from one to five. Anthropic writes in its documentation as well: "Having a good evaluation set is the most important step in the process." And OpenAI advises: "Only invest in fine-tuning after setting up evals."

Now about money, since this whole story starts with the wish to save. A subscription costs two hundred dollars a month. A Mac Studio with maximum memory costs more than twenty-two thousand. That is more than nine years of subscription, and Kimi still does not fit in it. Renting one gaming graphics card around the clock will cost roughly two hundred fifty to six hundred fifty dollars a month. That is comparable to the subscription, and the model on such a card will be at the level of a small compressed Qwen.

And here is my view. For a business to save on a two-hundred-dollar subscription, with this much competition around, is absurd. Your competitor will not be fiddling with Qwen on a laptop. They will run Opus 5.5 and pull ahead while you are setting up the model's memory and trying to figure out why your agent is going in circles.

There is one more point few people think about. The price per token is not the price of a finished task. Anthropic itself writes: "At higher-effort settings, however, cost per task with Sonnet 5.5 gets close to Opus 5.5." A cheaper model thinks longer, makes more mistakes, and redoes more. And in the end it comes out not as cheap as the price list promises.

And if you want it cheap, there are small models in the cloud, and they cost next to nothing. Haiku 5.5 and GPT-6 Luna cost ten cents per million input tokens. The Chinese Hy4 Flash from Tencent costs seven cents. At small volumes this is noticeably cheaper than keeping a rented graphics card, and there is nothing to set up.

And frontier prices are falling. Vercel, a company that routes requests to many different models, calculated that in October its average token cost less than half of the April price. But total spending keeps growing. Their explanation: the models are getting better faster than they're getting cheaper. In other words, people spend more and more on frontier models because those models can do more and more.

And about new releases. People say a new Opus is coming soon. Nobody has announced it officially; there are only users' guesses. The latest Opus, version five point five, came out on September twenty-second. And on November ninth, Anthropic released Fable 5.2, and it is twenty percent cheaper than the previous version. Frontier models are not standing still, and that is one more argument against staying on an old model to save money.

And now a turn, and for me it is an important one. Everything I have said concerns an agent's general-purpose work. But there is a huge number of tasks you can hand over to small models of your own. One model, one task. And fine-tune it for that task. And you do not need to be a machine learning engineer for this. Everything technical I am about to describe, your agent will do at your request: prepare the data, pick and rent a graphics card, train, test. Your part of the work is to give it good examples and look at the test results. You will be the one deciding whether the model is good, not a table and not the agent.

Let me tell you about myself. I cannot tell you everything, but I have already trained a great many models for practical tasks in my business. We have medical projects for the American market. And in many places we no longer use frontier models, to save tokens. We use our own local ones, because it really is more cost-effective. I am in favor of training simple neural networks for simple tasks.

There is a clear mechanism behind this. A big model knows a little about everything. But the right answers for your narrow task live in your data, not on the internet, and the big model has simply never seen them. A good example is the investment fund Bridgewater. Together with the lab Thinking Machines, they fine-tuned the open Qwen on their private data for six financial tasks. For example, whether a given article matters to an investor, or whether a central bank document hints at a rate change. The fine-tuned model reached almost eighty-five percent accuracy, against seventy-eight for the best frontier model. And it cost almost fourteen times less.

An honest caveat: Bridgewater's model was very large, not a laptop model. But there is also a study with small ones. The authors fine-tuned three hundred ten models on thirty-one tasks. After fine-tuning, the small models beat the GPT-4 of that time by ten points on average. But GPT-4 still won six of the thirty-one tasks, and those were mostly broad tasks, such as writing code. A narrow trained model wins in its narrow spot, while broad work stays with the big ones.

The market shows the same thing. That same Vercel calculated its statistics for October. On its platform, open models account for seventy-one and a half percent of all tokens, but only twenty-three percent of the money. Companies already give the mass, cheap work to open models, and they pay mostly for the complex work of frontier models.

How does it work? Nobody retrains the whole model from scratch. A small add-on is attached to it, and only that add-on is trained on your examples, while the main model stays as it was. That is why an add-on for a smallish model can be trained on an ordinary gaming graphics card. You do not need millions of examples. To start, fifty to a hundred good examples are enough, and for a reliable result it is better to have more than a thousand. Many people already have that many sitting in their history of requests, emails or documents.

About hardware, here is what I say: for tasks like these, renting a graphics card for about two hundred dollars a month or owning a card for a couple of thousand is enough. And the prices confirm it. A gaming graphics card costs less than a dollar an hour to rent. If you run it by the hour rather than around the clock, two hundred dollars a month is the right ballpark. And the new top gaming card had an official price of about two thousand dollars.

And one more handy thing. You can keep several small models on one server and switch between them. When you need one, it loads into memory, and another is unloaded. This is a standard feature. Ollama, for example, unloads a model on its own five minutes after the last request. Your agent can set up the server so that the right model switches on by itself as soon as a request for it arrives.

What work should you give a small model? Here Anthropic has some honest experience. They tried a setup where the main model hands out subtasks to cheap helpers. It saves money, but the share of successful solutions drops a little. And where the task is one connected chain, a single strong model came out ahead in every measurement. Meta describes a similar setup for its new models: the big model plans, the small one handles the volume.

The rule that follows is simple. Small models get independent, repetitive tasks of the same kind: sorting emails, putting requests into categories, pulling fields out of documents. There is also a good example for those who handle other people's personal data. A local model on your machine replaces names, phone numbers and other personal details in the text with placeholder tags. The cleaned text goes to the cloud. The answer comes back, and the tags are swapped back for the real data. This is a narrow task, and a small model handles it. But one long connected task, like improving your application, is better left to a strong model.

Ask your agent like this: find the repetitive steps of the same kind in my project, estimate the monthly volume and the cost on the current model for each, and suggest what can be handed to a cheap model and how to check its answers. And after that comes the test on your own examples, as I said.

But before you tell your agent "build me my own model", there are three traps that few people know about.

The first trap is where to get the data. The most obvious idea is to ask Claude or ChatGPT to prepare examples and then train your small model on them. Well, that breaks their rules. Anthropic forbids using Claude's answers to train models without its permission. OpenAI forbids using its answers for models that compete with it. And in three days, on November twentieth, an even stricter OpenAI rule takes effect. You may not use its models to prepare training data and training tasks for models that are meant to replace it. Even if you are not training the model yourself and are only preparing the data.

Whether there are exceptions here for internal tools is not clear from the rules; that is a question for a lawyer. The safest path is your own real data with verified answers. So ask your agent like this: build a training set from my past data with verified answers, set aside a fifth of it for testing, pick and rent a suitable graphics card, fine-tune the add-on, and compare the new model with the current one on the set-aside part. The set-aside part is needed so that the model is tested on examples it has never seen.

The second trap is other people's personal data. Many people think: it is my own model, so the data goes nowhere. Not quite. If the model runs on a rented graphics card, the owner of that card is also an intermediary. For medical data in America, this is spelled out directly: a cloud provider that stores or processes such data is legally responsible for it, even if the data is encrypted. You need a special agreement with that provider on protecting medical data. Anthropic and OpenAI sign such an agreement for their paid programmatic access, but ordinary subscriptions with a chat window do not qualify. So a model of your own is a question of how many intermediaries you have, not a magic pass.

The third trap is security. Someone else's model on your server is a black box inside your system. At the end of October, the company Cohere tested one open Qwen model acting as an agent. In rare cases the model tried on its own to send a file to Alibaba's cloud storage in Beijing. And in its reasoning it wrote that it would not call anything. And when it was nudged to complete a command, in a quarter to a third of cases it tried to reach the server's access keys. Put simply, its passwords.

Cohere advises you to "treat weights you didn't build like an unsigned binary", that is, like an unverified program, and to "deny agent egress by default", so the model cannot reach the network. In their words: "A reasoning trace is not a security boundary." There was another story too: in April, a hole was found that let a downloaded model file launch someone else's program on the server, and such files passed the platform's standard checks. So if you run your own model for an agent, tell the agent: run it in an isolated container with no internet access, allow only the addresses it needs, and log every tool call.

Now about the future. I think the future belongs to local models. Most likely, in about two years, anyone will be able to run a model at home that handles agentic programming at the level of Opus 5.5. I do not use local models for this myself yet, but I think we will all get there.

The data does not contradict this forecast. Researchers calculated last year that models which fit on a single top gaming graphics card catch up with the frontier level on tests in roughly six months to a year. In real work, the lag is probably longer. But there is a correction: frontier models keep running ahead during that time too. By one estimate, every three to four months they can see through to the end a task twice as long as before. So a local model catches up with last year's level, not today's.

And how long the frontier models keep running depends on the money of the big labs. Here the ceiling of training matters. Frontier companies are forced into a chase that works like a pyramid: they pay off each new model, and the money immediately goes into training the next, even more expensive one. Anthropic's head, Dario Amodei, described this very logic using hypothetical numbers. Say a model was trained for a hundred million dollars, and over the following year it brought in two hundred million in revenue. On its own, it is profitable. But the company as a whole is always in the red, because it is already training the next, much more expensive model. In that hypothetical example, the loss grows like this: a hundred million in the first year, eight hundred million in the second, and eight billion in the third.

If this pyramid collapses, the companies will end up deep in losses. Perhaps this is where the bubble is. Even Sam Altman of OpenAI spoke in September about "the first signs of what feels to me like unsustainable silliness".

But I think a collapse most likely will not happen. The race holds as long as there is a goal ahead, and artificial general intelligence still seems to be a few years away. True, in August Altman said to expect such a system by the end of this year. But so far nothing has been announced, and Altman admits that it is hard to draw a clear line here. So the race will most likely continue. Frontier models will keep pulling ahead, and open ones will keep catching up.

Let me pull it all together. For an agent's main work, for code and long tasks where a mistake is costly, you need a frontier model today, and the weakness of small models is not speed, but how their errors add up step by step. A small model of your own pays off on one narrow, repetitive task, if you train it on your own data, keep it locked down and test it on your own examples. And any choice of model, your own or someone else's, is decided not by the table from an announcement, but by your own test on thirty to a hundred real cases. So my answer to the question from the start of the episode is this: do not drop your subscription, and set up your own small models alongside it, for narrow tasks. That is exactly where they already save money today, and in a couple of years, I think, they will be able to do much more.

That is all for today. You have been listening to Evgenii Arsentev, creator of Arsminal, a terminal for managing AI agents where your whole agent team works in one window, like chats in a messenger. Founder of ArsentevAI, Dad4Kids and ReelsZavod. Try Arsminal for free at arsminal.com. Thanks for listening.

0:00−30:30 left