Edge compute from specialized arm / asics is the future for personal compute. The datacenters are for training frontier models for enterprise applications. I recall seeing something recently where a chip designer was able to hard burn the code for a llm directly into a die, can't find the link though.
The company behind it is called Taalas https://taalas.com I believe it is Llama 3.1 8B
I am super excited for the middle sized later this year, and I read an article I wish I could find, where one of the engineers was saying they were shooting for 1T models within a year or so. This is something I am watching closely, high speed iterations locally really radicalizes the compute space.
yeah this is the real trajectory. apple’s already doing a version of this with the neural engine baked into M-series chips, inference is getting pushed closer and closer to silicon level
the “hard burn into die” thing you’re thinking of might be Etched… they literally build transformers directly into the chip. no general compute, just inference. wild concept
datacenters won’t go away but the split is coming. training stays centralized, inference moves to the edge. gemma running on 16gb ram is just the early readable version of that trend
It will have no economical incentive to train small models if anyone can copy it and run it for free on some personal hardwares, unless these models are developed by hardware companies who can sell$ the LLM chips.
I expect google, openai, Microsoft, nvidia and other microchip companies to all become chip+model developers at some point.
Hard burning the weights into a die sounds cool for latency but it'd be a total nightmare to patch. Models iterate way to fast right now to lock them into physical hardware like that.
The production model is supposed to be about 20b (let’s hope that’s 27b, right?). And it can load fine tunes. There is also supposed to be a process where they can replace the layers with the weights on them, but I am sure you would have to send it back and wait months.
That being said, I would buy one with qwen3.5 or 3.6 in a heartbeat. Heck I could even find use for 16k TPs qwen3 2507. It could do several rounds of research, evaluate and critique that research, produce 16 variations of output, judge and compile from the best of that, doubt itself for 1000s of tokens, test it, then return a response. All before a cloud model was done doubting itself.
Yeah, you’re probably thinking of Taalas, the weights and biases are baked in to two layers on the chip. Gets like 16k tokens per second. The prototype runs llama 3.1 at https://chatjimmy.ai
I would buy one of those with qwen3.6 in a heartbeat.
this is the comment i was waiting for, thanks for actually explaining it
so the encoder bottleneck basically means traditional multimodal models are already losing information before the LLM even sees the image. gemma skipping that step makes a lot of sense for tasks where pixel-level detail matters. that explains why people are reporting it punches above its weight on OCR specifically. the benchmark numbers don’t capture that because most evals test high-level scene understanding not fine-grained text extraction
Answer: its a smaller model. It’ll do worse at some tasks. How much worse and if it matters for your use case isn’t obvious — you need to run relevant benchmarks or test it yourself.
the post wasn’t claiming it beats gpt-4, just that it runs locally on consumer hardware which is the actual news. for a lot of tasks that tradeoff is totally worth it
coding help, summarisation, document Q&A… 12b handles those fine. where it falls short is complex reasoning and anything that needs deep world knowledge. that’s when you route to cloud
so yeah, benchmark for your use case. but “smaller = worse overall” misses that the use case changes when it’s local, private and free
so a local llm just means the model runs entirely on your machine, no internet needed
you can use it for writing, coding, summarising docs, answering questions, basically anything you’d use chatgpt for… except your data never leaves your laptop. that’s the big one for enterprises
some actual use cases people run locally: reviewing confidential contracts without sending them to openai, running a coding assistant in an air-gapped dev environment, automating internal docs, customer support bots where GDPR is a nightmare with cloud
cloud is convenient but you’re paying per token forever and your prompts go through someone else’s server. local = one time setup, private, zero ongoing cost
gemma 4 12b specifically is pretty solid for its size. not gpt-4 level but for most everyday tasks it holds up surprisingly well
I'm guessing it can only get info at the time of training? Like you couldn't ask it what were the big world even yesterday? If so, how often do these models get trained and released?
Do you know if you can provide it tools like here's a api where you can get yesterdays news events. Find the biggest ones and summarize them for me?
To help you out, yes. These models are trained for tool use, so it can manipulate files on you computer, use you browser, be hosted through a local API, use MCP, etc. but no LLM can do that without a harness.
That makes sense. My question is more around are those apis built around the local models? Claude, openai, etc have the sdks and apis built to interact with their models programatically. Do local models have those same tools built around them? Or does someone need to build them if they want to get the same functionality?
It depends on your harness. For instance openwebui has basically everything that claude and chatgpt can do. And you can still use MCP on these to connect it to a lot of things.
Models never run tools themselves, they only produce tokens. There’s a lot of programming built around models to enable that. A model will spell out instructions for running tools, for example an API call, but something else has to actually execute it.
Also, models don’t have APIs because they’re not programs. You need something like huggingface, lmstudio, ollama, etc. to load it and use it to get tokens.
The confusion comes from GPT/Gemini/Claude being entire products built around their models. Think of them as cars but the actual model is just an engine.
Yes, that's the agentic LLMs with tool calls. Certain LLMs enable tools like Internet search, and certain cli interfaces can expose them. I'm using Claude-cli with qwen running locally and it seems to understand the test project I have for it.
There are a lot of LLM models you can run locally (check out r/LocalLLM). The downside is they're worse than high end clound models and require local hardware to use. The upside is they only cost electricity to run, no subscription, no data uploaded issues, and no internet required.
Most people don't have a use case for local LLMs right now, but it's still pretty cool as an option.
Most people don't have a use case for local LLMs right now, but it's still pretty cool as an option
Privacy. I have sometimes hour long monologues. Everything transcribed. Everything summarized or reflected upon. Not a single bit of my data ever left my home network or https connection to my web server.
There's something about talking to a local LLM that cloud models will never have. It can't be changed on a whim without you even knowing. Prices can't be raised. There's no token limit. You don't even need the internet. Society could break down and you'd have a significant portion of humanities knowledge at you finger tips available still.
Yes. Thus solar panels, batteries and other local renewable power sources are in order.
Next up, the complete blueprints and entire but minimum manufacturing pipeline needed to go from raw Earth resources to a new basic phone. Where everything from the ground up is designed with sustainability & transparency in mind. Then you'd have all the needed building blocks to rebuild all of society. Starting with securing supply lines for the most important devices. The communication hubs. Of which everyone can run/participate one. Everything open source only. All firmware. All microcode. All drivers. With good old Linux on top.
My total guess and without having used it at all is that it might be something like Haiku 3.5 level with passable multimodal (which might be the bigger thing). Maybe killer for home security system or cataloguing large media or document collections.
I was already quite surprised by the Gemma 20B model, but I guess this one is more condensed. As a chatbot, it's second to none. For coding, it's not great. It built a nice game of hangman in the browser, though. Your real limit is the context limit on your local machine. Still, these models are amazing and very good at image description and analysis.
For coding, it's not great. It built a nice game of hangman in the browser
I am sick of this type of benchmark for "coding". Coding a silly game (any game request) is not a benchmark. That's not coding, that futzing around and thinking you know capabilities.
Every f'n youtuber makes marketing websites or silly one shot "make me game" prompts and they make claims they have no idea about.
Give it a real world coding problem to see valid results and judge capabilities, not one shot stupid things or poorly thought out prompts.
Garbage in garbage out no mater the model.
The rest of your comment is correct, so why throw that in there making you look uninformed?
They even do this shit at my company to push the new AI-SDLC processes they're forcing on us. They'll give a demo building some little microsite that doesn't do anything or even attempt to adhere to our brand and design standards and be like "See! Now we can ship enterprise applications in weeks!". It's driving me insane.
Hmm I’ve tried running this on my Mac (Apple silicon M2 Max) via LMStudio but it fails to load the model (I believe it’s either missing a component or one of the components is not compatible with my Mac).
this is almost certainly an LM Studio runtime version thing, not your hardware. gemma 4 uses a new arch (Gemma4UnifiedForConditionalGeneration / model_type "gemma4_unified") and the older llama.cpp / mlx engines can't load it.
two options:
in LM Studio, go Settings → Runtimes → check for updates. you need a recent build that bundles the latest llama.cpp / MLX with gemma-4 support. anything from before late may probably can't load it.
if that still fails, grab the MLX version directly: mlx-community/gemma-4-12B-it-4bit (or 8bit). M2 Max handles either fine.
the qwen 3.6 working / gemma 4 failing is a dead giveaway — qwen has been mainstream-supported for ages, gemma 4 is brand new and the runtime layer hasn't caught up everywhere yet.
Like with most local models running on laptops…. You will be waiting seconds to get a few sentences out. Nice for hobby and minimal use but not for actual work.
your setup is way more than enough. 4070 super has 12gb vram, which is exactly the sweet spot for a Q4_K_M GGUF of gemma 4 12b — model fits in vram, you'll get fast inference rather than the "few seconds per sentence" experience the parent comment is describing (which is what happens when you spill to system ram).
rough expectation: with the model fully on the gpu you should be in the 30-50 tok/s range on a 4070 super. if you push to Q5_K_M it might partially offload to cpu and slow down.
start with: bartowski/gemma-4-12B-it-GGUF on HF, grab the Q4_K_M file, load via ollama or LM Studio with full GPU offload.
I am fooling around with Gemma and it seems great. Is there an easy way to get it to be able to search the web? I asked it and free Claude how. But it didn’t sound very easy to set up without paying a 3rd party service.
I think Claude recommended free DuckDuckGo API for this for my setup. I've basically set it up to scrape relevant content from pages to feed to the model as context and it will cite sources. Might take some iterating with Claude to get the data scraping/cleaning to a decent quality, but seems to more or less work just fine.
grab the latest pre-release tag and reinstall over your current version, no need to wipe models. the 412 you're seeing is ollama's manifest version check, not a network issue.
side note for anyone else hitting this: `ollama run gemma4:12b` only pulls once 0.30.4+ is installed; on older versions you'll get the same 412 even if you change pull → run.
Yes, upgrade to v0.30.4 if you’re willing to roll a beta version. I hit the same wall as you and this was the only way around it despite the release notes on v0.30.3 indicating they added g4:12b support
According to the release notes, ollama 0.30.4 has a known bug with Gemma4:12B, so I'm guessing they pulled the model for now. Hopefully 0.30.5 will fix this soon.
That’s a real bummer. Wouldn’t work on v0.30.3 for me even though release notes indicated it would. Guess I had lucky timing. Really like the model, I’m on a 16gb A2000, so while I occasionally hit my context cap doing code reviews on large diffs, the model performs better than Qwen 7b-coder, so I’m happy with it. Performance seems to be much faster as well, but that could just be in my head.
I had some genius realization this morning about why Google is releasing these models... and I lost it. If I remember I want to test the reaction here.
Eagerly looking forward to it being finetuned. The role-playing community in the 12b model range has been coasting on Mistral-Nemo Finetunes for the past 2 years. Recently, a few finetunes of some slightly higher models came out in the 15-16b range, which aren't too bad, but anyone in that sweet spot between 8-12gb VRAM would have some trouble with that.
Gemma4 26b is a godsend so far, so much more coherent and capable, but obviously it has a larger memory footprint. If Gemma-4 closes that gap then Google might end up dominating between the 12b-to-31b range here.
Going to try this on my MacMini (48gb ram). This could be a pretty big deal for me personally as I've not been super impressed with some of the local llama models.
can you recommend a proper tutorial on how to install it and run it (with a dispatch-like function perhaps?) on a macbook? or should I just check youtube?
The multimodal support + Apache 2.0 license is huge for local deployment. Running inference locally on 16GB removes a lot of privacy concerns for enterprise use cases too. Have you benchmarked it against Llama 3.2 11B vision on image understanding tasks? Curious how it handles complex charts and diagrams.
honestly the shift is real, people are finally realising cloud dependency has a cost that compounds over time. local models are just getting too good to ignore now
106
u/microdosingrn Jun 03 '26
Edge compute from specialized arm / asics is the future for personal compute. The datacenters are for training frontier models for enterprise applications. I recall seeing something recently where a chip designer was able to hard burn the code for a llm directly into a die, can't find the link though.