Open Models: MoE, Reasoning & Fine-tuning

Part 1 · Open Models

The open-weight landscape

"Open" is not one thing — it is a range. At one end, a project releases everything: the weights, the training data, and the training code, under a license that lets anyone use it freely. At the other end, a company releases only the weights, under its own custom license with rules attached. That second, more common case is what people mean by "open weights." When you pick a model to run, four practical questions matter: does the license let me use it in a commercial product, what design does it use (explained below), how much text can it read at once (its context window), and what is it especially good at.

Two design words appear in the table. A dense model runs every one of its weights on every word — simple and predictable. A Mixture-of-Experts model (MoE for short, explained fully in Part 2) stores many weights but uses only a small slice of them on each word, so it is cheaper to run than its size suggests.

1.1 Why models end up good at different things

One family is best at writing code, another at chatting in many languages, another at using tools on your behalf. These strengths are not accidents. They are decided at three stages, all built on top of the same basic transformer:

• What it reads during first training (the pre-training data mix): a coding model is fed trillions of words of source code and technical docs; a generalist is fed a balanced diet of web pages, books, and text in many languages.
• How it handles long inputs: models that can read very long documents use tricks in how they track word positions (named RoPE scaling and YaRN) to avoid the known problem where a model ignores facts stuck in the middle of a long input.
• What it practices after first training (post-training): a tool-using model is trained on recordings of real tool calls — the JSON requests, the API calls, the results — while a chat model is trained on examples of people ranking which reply is better.

Part 2 · Open Models

Mixture-of-Experts, in full

Almost every very large open model uses one trick to stay affordable to run. The trick answers a question that sounds impossible: how do you get the knowledge of a 671-billion-weight model while paying to run only about 37 billion weights per word? The trick is called Mixture-of-Experts, or MoE ("mixture of experts" — a set of specialist sub-networks the model chooses between).

2.1 What an "expert" really is

Recall the transformer block from the Transformer post: first an attention step that lets each word look at the others, then a feed-forward network — a small stack of math that transforms each word on its own. That feed-forward network is usually written FFN (feed-forward network). In a normal dense model there is one big FFN in each block, and every word passes through all of it. In an MoE model, that single FFN is replaced by many smaller FFNs sitting side by side. Each one of these smaller FFNs is called an expert. So "expert" is not something clever — it is just one FFN out of many.

A small extra piece decides which experts each word should use. It is a tiny layer that scores every expert and picks the best few. It is called the router. The attention step is left shared and untouched — it keeps the flow of context between words — and only the FFN part is split into experts.

2.2 The routing math

Here is what actually happens for one word. The word arrives as a list of numbers, written \(x\). The router multiplies it by its weight matrix \(W_g\) to get one score per expert. It keeps the highest \(k\) scores (the "top-k"), turns those scores into fractions that add up to 1 using softmax, and produces the final output as those fractions times the outputs of just the chosen experts:

Each expert \(E_i\) is just an ordinary FFN, \(E_i(x)=\text{Activation}(xW_{i,1})W_{i,2}\). "Top-k" means keep the k best-scoring experts; the softmax turns their raw scores into weights that sum to 1 so the combination is a proper weighted average. For example, Mixtral has \(N=8\) experts and picks \(k=2\) of them per word: about 47B weights in total, but only about 13B doing work on any given word.

2.3 The problem MoE creates: keeping the experts balanced

If you just let the router learn freely, it cheats. It discovers a few experts it likes and sends almost every word to them, so the rest of the experts never get used or trained. Those unused experts are called dead experts. The general problem of keeping traffic spread evenly across experts is called load balancing. Two fixes are standard:

• A balancing penalty added to training (the auxiliary load-balancing loss, from the Switch Transformer work): an extra term in the training goal that punishes lopsided expert usage across a batch of words, pushing the router to share the load.
• Balancing without a penalty term (the auxiliary-loss-free method, from DeepSeek-V3): instead of a penalty, it nudges a small per-expert offset up or down to even out traffic, so the main training goal is not distorted — a newer, cleaner approach.

DeepSeek adds two more ideas. Fine-grained experts means using many small experts instead of a few big ones, so each can specialize more narrowly. Shared experts means keeping one or two experts always turned on for every word, to hold the common knowledge every word needs, which frees the routed experts to specialize harder.

Part 3 · Open Models

Reasoning models — training a model to think first

A plain language model just predicts the next word that is most likely to come — a fast, one-shot guess. For hard logic problems that quick guess is often wrong. A reasoning model is trained to write out a long train of thought before it gives its final answer, checking and correcting itself as it goes. It moves effort from training time to answer time — the time when it is actually producing your reply, which people call test time.

3.1 Thinking out loud, as a trained habit

The model writes its working-out inside special marker tokens, <think>...</think>, and then writes the final answer after. Why does writing out its thoughts help? Because of a fact from the foundations: each word the model generates becomes part of the input for the next word. So when it writes "step 1… step 2… wait, that is wrong, let me redo it…" it is literally giving itself more chances to compute and more notes to build on before committing to an answer. This is called chain-of-thought — the model reasoning in a visible chain of steps. More thinking words means more work done at answer time, which means better accuracy on math, code, and logic — up to a point, after which extra thinking stops helping.

3.2 How the thinking is trained: rewards for getting it right

DeepSeek-R1 showed you can teach a model to reason mostly by trial and error with rewards, a method called reinforcement learning. Instead of copying human-written reasoning, the model tries to solve problems whose answers can be checked automatically — a math problem with a known result, or code that must pass a test. When it solves one, it gets a reward. Over many tries it discovers, on its own, that checking its steps and going back to fix mistakes gets more rewards.

3.3 Hybrid reasoning — one model with a thinking dial

Early on, a reasoning model was a separate download from the normal model. Newer models (Qwen3, Claude, and others) combine both in one and let you set how much thinking you want. This combined kind is called a hybrid reasoning model, and the setting is a thinking budget — a cap on how many thinking words it may use. Set the budget to zero and it answers instantly like a normal model; raise the budget and it works through the problem first.

Part 4 · Open Models

Quantization & precision — making a model fit

Every weight in a model is stored as a number with a certain number of digits of accuracy. The more digits, the more memory it takes and the slower it is to move around. Storing each weight using fewer bits — trading a little accuracy for a lot of memory savings — is called quantization. It is what lets a model that supposedly "needs" a data-center GPU run on a laptop.

4.1 The memory math

The memory a model's weights take is simply the number of weights times how many bytes each weight uses:

The precision level tells you the bytes per weight. Some names you will see: FP16 and BF16 are two 16-bit (2-byte) formats — "full" precision for running a model. INT8 stores each weight as an 8-bit (1-byte) whole number. INT4 and NF4 use just 4 bits (half a byte) each. Here is the same 8-billion (8B) and 70-billion (70B) weight model at each level:

Check the math: 8 billion weights × 2 bytes = 16 GB at FP16; at 4 bits (half a byte) it is 8 billion × 0.5 = 4 GB. Two things get cheaper at once. First, capacity: the model fits in less VRAM. Second, speed: producing words is limited mostly by how fast weights can be moved from memory into the compute cores, and 4-bit numbers move four times faster than 16-bit ones. So a 4-bit model is about a quarter of the size and noticeably faster, at a small cost to quality.

4.2 The formats you will actually run into

Each name below is a different way of storing or producing quantized weights. Here is a one-line plain description of each:

Part 5 · Open Models

Fine-tuning: LoRA & QLoRA on your own GPU

Adjusting a base model so it fits your domain, your tone, or your specific task is called fine-tuning. The thorough version — updating every weight in the model — is called full fine-tuning, and it needs a cluster of data-center GPUs, so most teams never do it. The technique that made fine-tuning affordable for everyone is LoRA, and a further trick called QLoRA lets you fine-tune a capable model on a single gaming-grade GPU.

5.1 LoRA — train a tiny add-on, freeze everything else

Instead of changing a model's giant weight matrix \(W\) directly, LoRA (which stands for Low-Rank Adaptation — "low-rank" is explained in a moment) leaves \(W\) frozen and learns a small correction to add on top. It writes that correction \(\Delta W\) as the product of two thin matrices, \(B\) and \(A\):

Only \(A\) and \(B\) are trained; the huge \(W\) never moves. The number \(r\) is the rank — how thin the two matrices are — and "low-rank" just means \(r\) is chosen much smaller than the matrix's normal dimensions. Concretely: take a \(4096\times4096\) weight matrix, which has \(4096\times4096\approx16.7\) million weights. With rank \(r=16\) you instead train \(B\) (size \(4096\times16\)) plus \(A\) (size \(16\times4096\)), which is \(2\times16\times4096\approx131{,}000\) numbers — about 0.8% as many — and still recover most of the quality of full fine-tuning. The scalar \(\alpha/r\) out front is just a knob for how strongly the add-on is applied.

5.2 QLoRA — LoRA on top of a 4-bit base

QLoRA (Quantized LoRA) goes one step further. It loads the frozen base model in 4-bit NF4 (the 4-bit format from the last section), then trains the small LoRA adapters on top of that in full 16-bit precision. The big part — the base weights — is quartered in memory, while the small trainable part stays at high accuracy. This is what brings fine-tuning a 7B-to-14B model down to a single high-end consumer GPU such as an RTX 4090.

5.3 The four steps, in code

# 1 · Load the base model in 4-bit NF4
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct",
                         quantization_config=bnb, device_map="auto")

# 2 · Attach the LoRA adapters (only these get trained)
from peft import LoraConfig, get_peft_model
model = get_peft_model(model, LoraConfig(
    r=16, lora_alpha=32, target_modules=["q_proj","v_proj"],
    task_type="CAUSAL_LM"))

# 3 · Train on your data (each example wrapped in the model's chat template!)
from trl import SFTTrainer
SFTTrainer(model=model, train_dataset=ds,
           dataset_text_field="text", max_seq_length=2048).train()

# 4 · Save just the tiny adapter (tens of MB, not tens of GB)
model.save_pretrained("./my_adapter")

Part 6 · Open Models

Alignment: DPO & ORPO after SFT

Training a model by showing it good answers to copy is called supervised fine-tuning, or SFT. It teaches the model to imitate. There is a further step called preference tuning: instead of one right answer, you show the model two answers and tell it which one is better. This captures subtle wishes like "be helpful but keep it short" that are far easier to demonstrate by comparison than to spell out in a rule.

6.1 From RLHF to DPO

The classic way to do this, from the frontier post, is RLHF — Reinforcement Learning from Human Feedback. It first trains a separate model to score answers (a reward model), then uses reinforcement learning (the PPO method) to push the main model toward high-scoring answers. It works, but it is a fragile pipeline with several moving parts. DPO (Direct Preference Optimization) does the same job in one shot: given pairs of answers — a chosen one \(y_w\) and a rejected one \(y_l\) — it adjusts the model directly using a single, simple loss, with no separate reward model and no reinforcement-learning loop.

In plain words: raise the model's probability of producing the chosen answer and lower its probability of producing the rejected one — but measure both against a frozen copy of the original model (written \(\pi_{\text{ref}}\), the reference) so the model does not drift too far from where it started. Here \(\pi_\theta\) is the model being trained, \(\beta\) controls how firmly it pulls, and \(\sigma\) is the logistic function that turns the score gap into a value between 0 and 1. ORPO (Odds-Ratio Preference Optimization) goes further still: it folds the preference step into ordinary SFT so it is one stage, with no reference copy needed at all.

6.2 The tooling

You almost never write these methods from scratch. Hugging Face PEFT is the library that provides LoRA and QLoRA adapters. Hugging Face TRL provides ready-made trainers for SFT, DPO, and GRPO. Unsloth is a library that makes training 2–5× faster while using less VRAM. Axolotl and LlamaFactory wrap the whole process so you drive it from a config file instead of code. The tooling is mature enough that the code is no longer the hard part — the hard parts are the data and the evaluation.