Gemma vs Qwen: Which Local AI is Best?

Image: Dan Ackerman
Replacing ChatGPT, Claude, or Gemini with a local AI isn't as difficult as you'd think. Actually, it's surprisingly easy. For power users, especially those that work with sensitive documentation, it's worth considering offloading a good portion of AI workload to a model that sits privately inside your computer. For some, it's a better alternative to sending data to server farms in Nevada. The way I see it, if you really need more computational horsepower, ChatGPT is only a click away, right?
That's the conclusion I came to after testing Qwen3.5 by Alibaba Cloud and Gemma 4 by Google. While there are other local models available, I felt these two were both popular and recent enough to be worthy of a versus. I honestly don't see people opting for an older model like DeepSeek R1 when newer models exist.
[Editor's note: Qwen 3.6 has just been released and we are currently testing it.]
To my surprise, not only were both models competent, they felt fairly competitive against major cloud models that cost $20/month. To me, it felt like using OpenAI's o3 model. This was a reasoning-capable model released in mid-2025 that has since been discontinued. Of course, feeling like o3 and performing like o3 are two different things.
Can local models that run under your desk outperform ones running in giant multi-billion dollar data centers in Texas? Definitely not. But the models have seen recent optimizations that deliver strong performance without ever needing an internet connection. (Although, it is possible to connect your local models to the internet.)
The major drawback to local AI models, however, is their lack of connection to the outside world out of the box. Since both Qwen3.5 and Gemma 4 are trained on data up until a certain date, you won't be able to ask timelier questions. In these cases, turn to Google Search or an online AI. The local models I tested can't generate images, either. Given the computational overhead, it's probably still best to turn to cloud models for image and video generation.
So, which local AI model is best?
Between Qwen3.5 and Gemma 4, I found Google's model to perform in a way that was most satisfying to me. What I mean by this is that it didn't necessarily beat Qwen3.5 in all the questions I asked. Actually, Gemma 4 was more prone to making errors or hallucinations. But in actual use, it was easier to deal with.
Response time on Gemma 4 is significantly faster. Even a simple "hi, how are you?" takes much more time on Qwen3.5 to respond. When trying to do larger workloads, I'd find myself browsing Instagram just waiting for Qwen to respond. I am running both models at a Q6 quantization, but I'll elaborate on that further later on. Honestly, given how closely both models performed in my tests, picking the faster model, in my opinion, is the way to go.
Here's my AI rig
There are plenty of models available online that can run on all sorts of configurations. For this test case, I built an incredibly powerful PC with a $4,000-plus price tag. You don't need to spend this much to run AI on your computer but systems with more memory can run larger models with greater accuracy and abilities.
- AMD Ryzen 9 9950X3D
- Nvidia RTX 5090
- MSI MPG B560I Edge Wi-Fi Motherboard
- 32GB Crucial Pro DDR5 RAM
- 2x 1TB Crucial Gen5 NVMe SSDs
- Corsair SF1000 Power Supply
- Fractal Design Terra Mini-ITX Mini Tower
- Noctua NH-L12S CPU Cooler
The focal point of this build is the Nvidia RTX 5090, with its 32GB of GDDR7 VRAM. With this much VRAM, I'm able to run large 35-billion parameter models that are over 25GBs in size without too much issue. If, for example, you have a RTX 5070 Ti with 16GB of VRAM, it'd be best to pick an AI model that's under 16GB in size. VRAM will largely determine the size of model you can load. Some of that can be offloaded to system RAM, but you may run into issues of the AI model tapping out.
If you have the cash, going with the Nvidia DGX Spark is a great option. It's like a tiny AI server you can connect to your computer. But with 128GB of RAM, it can handle much larger models.
Which specific model variants did I test?
I tested specific variants of Qwen3.5 and Gemma 4. For Qwen 3.5, it was the 35-billion parameter model at Q6-level quantization with three billion active parameters (A3B). One way to look at this is Qwen's entire library of knowledge is 35 billion books, but this local version opts to use three billion of the most prescient titles to answer questions. This is called the Mixture-of-Experts (MoE) architecture. The Q6 quantization refers to the compression. For example, if I opted to use Q4, the model would have been lighter, but less precise. And conversely, if I had gone for Q8, the model would have been heavier, but might not have run well on my PC. For the specs on my rig, Q6 was the sweet spot.
With all that in mind, the version of Gemma 4 I used had 26-billion parameters also at a Q6-level quantization with four billion active parameters (A4B). So, while Gemma 4's overall library was smaller, it pulled out a larger number of books to answer questions. Google did release a heavier 31-billion parameter model, but for the purposes of this comparison, I'll stick with the 26-billion parameter one.
How I review and test local AI models
AI chatbots are machines that can do a bit of everything. For that reason, it's impossible to truly exhaust all variables when testing AI models. I've found the best course of action is to do a variety of tests that aim to replicate real-world usage. Sure, I can try running benchmarks like GDPval and Humanity's Last Exam, but people generally aren't using AI models in this way. Plus, there are plenty of sites out there that put AI models through the benchmarking ringer. For this guide, real-world usage is best. Although, if you don't want to read through my evaluation, please, feel free to check scores on Artificial Analysis. There are also some interesting evaluations from users on Reddit that follow more benchmark-like testing.
The questions I ask range from general inquiries, like travel or restaurant recommendations, instruction tests, article or essay writing, vibe coding and everyday chit-chat. The conclusions in this comparison are very much my own.
Why Gemma 4 beats Qwen3.5
While Gemma 4 did technically make a few more hallucination errors than Qwen3.5 in my testing, the overall experience of using Google's model was better. The speed at which Gemma 4 answered my questions made it feel like an online chatbot. That means faster output for you, and potentially cheaper overhead if you plan on running models overnight.
When it came to essay writing, Gemma 4 read more "human," being less prone to constant metaphors, a quick giveaway of AI-generated content. That's not to say responses read completely human. It's just that Qwen3.5 definitely had that AI feel. Both certainly did love the use of em-dashes, however, a hallmark of AI-written content.
In my riddle test, both Gemma 4 and Qwen3.5 handled it with ease, answering questions correctly or finding the logical errors. In college essay writing, both were able to cite evidence with links that actually worked. In Gemma 4's case, however, it hallucinated a source whereas Qwen3.5 didn't. Is that the end of the world? I don't think so. There's a degree of oversight any user of AI must adhere to, right?
In another writing test, I told both models to write me a 500-word article in the style of CNET. Both models did a good job of emulating CNET's consumer-facing style, but neither could follow instructions, hitting that 500-word mark. Gemma 4 got closest, however.
That's not to say Qwen3.5's answers were perfect. When recommending restaurants in the Houston area, it told me to go to Whataburger. Bleh. There are far better burger spots (and In-N-Out) in Houston. Surprisingly, Gemma 4's restaurant recommendations were solid throughout. I assume Gemma 4 had access to Google Maps training data.
Additionally, both models can do visual analysis. In this test, I uploaded a picture of an old bathroom photo I found on Reddit and asked each model to give renovation recommendations with a $3,000 budget. Surprisingly, both models did an excellent job discerning the colors, fixtures and even materials. Both suggested removing the linoleum tile for luxury vinyl flooring, to remove the dated light fixture at top, and to swap out the drab colors for more muted tones. Honestly, when it came to visual analysis, I'd call it a draw.
Vibe coding: surprisingly good, but cloud models are better
Local AI models have gotten better at vibe coding with Qwen3.5 and Gemma 4 outperforming their past iterations. I had both models create a web-based Breakout-style game that could be played with a mouse. Both results worked immediately, albeit with some quirks.
Between the two, Gemma 4 created the most polished product and was easier to give recommendations for further improvement. Qwen3.5 needed additional prompting to make a workable final product, and even then, still didn't have the same logical reasoning as Gemma 4. For example, for Breakout-style games, the ball has to first start off slow and increase with speed. Qwen3.5 almost felt lightspeed at first and took some prompting to get it to an acceptable level.
Do note: I myself can't code, so I wasn't diving in and looking at the quality of the lines. The prompts I gave weren't complex or highly detailed either. For this reason, take my findings here with a degree of skepticism.
Closing thoughts
The AI world is moving quickly. While working on this evaluation, Google released a more capable 31-billion parameter model. It's possible that this version outperforms Qwen3.5 on all fronts. In my preliminary testing Gemma 4 31B is much slower, likely because of how it processes tokens at inference when compared to 26B. Still, when considering day-to-day usability, I found Gemma 4's 26-billion parameter model to perform almost identically to Qwen3.5's 35-billion parameter model while being significantly faster in the process.
If time isn't a concern, then there's a chance that Qwen3.5 (or Gemma 4 31B) might be the better model for you. It might be particularly tuned for more complex academic research, for example. Still, when evaluating these models, I have to consider them in totality. And when it came to writing quality, speed, visual analysis, and vibe coding, Gemma 4 took an ever-so-slight edge.
More from MC News
- Run AI Locally: The Best LLMs for 8GB, 16GB, 32GB Memory and Beyond
- Quantization Explained: Why the Same LLM Gives Better Results on High-End Hardware
- Why VRAM and Memory Bandwidth are Key for Powering Local AI
- Why AMD's 3D V-Cache is Having a Moment
- The Micro Center Monitor Buying Guide: Part 1
- Keyboard 101: Intro to Computer Keyboards
- Fix It Yourself: Talking to iFixit on Why Repairable Tech Matters
- Hands-on with NVIDIA DGX Spark: Everything You Need to Know
Imad Khan is a journalist based in New York. He covers AI, Big Tech, internet culture and gaming. His bylines include CNET, The New York Times, The Washington Post, ESPN, Tom's Guide and Wired, among others. When he's not chasing stories, he's either playing his guitar, working on custom Game Boys or falling down some nerdy YouTube rabbit hole.
