Making of an SLM

Musings Public · Protected · Private

Making of an SLM

Type: Public | Created: 2026-05-05 | Frozen: No

« Previous Public Blog Next Public Blog »

Comments

An SLM is an AI model trained to understand and generate text — just like GPT-4 — but with dramatically fewer parameters (1M to 7B). This makes them fast, cheap, and runnable on everyday hardware.
SLM Parameters
1M – 7B
Fits on a laptop or phone
LLM Parameters (GPT-4)
~1.8T
Requires a GPU datacenter
SLM Inference Cost
≈ $0
Runs locally, no API fees
LLM Inference Cost
$$$
Per million tokens billed
Simple Analogy
Think of an LLM as a professional encyclopedia — knows everything but lives in a library. An SLM is like a pocket handbook — knows what you need, fits in your bag, always with you.

2026-05-05 18:03

7 Ways to Create an SLM

Each approach has a different starting point, cost, and target use case.

Train from Scratch

Full control — you design everything

Design a compact transformer (fewer layers, smaller embeddings) and train it on a curated corpus using next-token prediction.

Simple Example

Python

— Tiny Transformer from scratch

import torch
import torch.nn as nn

class TinySLM(nn.Module):
    """A minimal 2-layer transformer SLM."""
    def __init__(self, vocab=5000, dim=128, heads=4, layers=2):
        super().__init__()
        self.embed   = nn.Embedding(vocab, dim)
        self.pos_enc = nn.Embedding(512, dim)           # positional
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=dim, nhead=heads,
            dim_feedforward=dim*4, dropout=0.1, batch_first=True
        )
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=layers)
        self.head = nn.Linear(dim, vocab)

    def forward(self, x):
        positions = torch.arange(x.size(1), device=x.device)
        x = self.embed(x) + self.pos_enc(positions)
        x = self.transformer(x)
        return self.head(x)          # logits over vocabulary

# Create the model — only ~1M parameters!
model = TinySLM(vocab=5000, dim=128, heads=4, layers=2)
total = sum(p.numel() for p in model.parameters())
print(f"Parameters: {total:,}")   # ~1,300,000

# Training loop (simplified)
optimizer = torch.optim.Adam(model.parameters(), lr=3e-4)
loss_fn   = nn.CrossEntropyLoss()

for step, (x, y) in enumerate(dataloader):
    logits = model(x)
    loss   = loss_fn(logits.view(-1, 5000), y.view(-1))
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if step % 100 == 0:
        print(f"Step {step}, Loss: {loss.item():.4f}")

Pros

Full architecture control
No licensing restrictions
Tailor to domain from day 1

❌ Cons

Needs large dataset
High compute cost
Months of training time

2026-05-05 18:04

Knowledge Distillation

Teacher LLM → Student SLM

Train a small student model to mimic the probability distributions of a large teacher model. The student learns "soft" knowledge, not just hard answers.

Loss = α · CrossEntropy(predictions, true_labels) + (1−α) · KL_Divergence(teacher_probs ‖ student_probs)

Simple Example

Python

— Distillation training loop

import torch, torch.nn as nn, torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, true_labels,
                       temperature=4.0, alpha=0.7):
    """
    temperature : higher = softer teacher distribution
    alpha       : weight of distillation vs hard-label loss
    """
    # Soften both distributions with temperature
    soft_teacher = F.softmax(teacher_logits / temperature, dim=-1)
    soft_student = F.log_softmax(student_logits / temperature, dim=-1)

    # KL divergence (distillation loss)
    kl_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean')
    kl_loss *= (temperature ** 2)          # scale back

    # Standard cross-entropy (hard labels)
    ce_loss = F.cross_entropy(student_logits, true_labels)

    return alpha * kl_loss + (1 - alpha) * ce_loss

# Usage
teacher.eval()
with torch.no_grad():
    teacher_out = teacher(input_ids)           # big model, frozen

student_out = student(input_ids)               # small model, training
loss = distillation_loss(student_out, teacher_out, labels)
loss.backward()
optimizer.step()

Real-World Example

DistilBERT — distilled from BERT-large (336M params) to 66M params. It runs 60% faster while retaining 97% of BERT's accuracy on GLUE benchmarks.

✅ Pros

Student learns reasoning patterns
Works for any task

❌ Cons

Needs access to teacher
Distillation itself needs GPU

2026-05-05 18:07

Model Pruning

Cut the fat from a big model

Remove weights, neurons, or attention heads that contribute least to model output. A 70B model can shed 40–60% of weights with minimal accuracy loss.

Pretrained LLM

→Identify unimportant weights

→Remove / zero them

Fine-tune to recover

→

Smaller SLM

Simple Example

Python

— Magnitude pruning with PyTorch

import torch
import torch.nn.utils.prune as prune

# --- Structured pruning: remove entire neurons ---
def prune_model(model, amount=0.4):
    """Remove 40% of weights by magnitude across all Linear layers."""
    for name, module in model.named_modules():
        if isinstance(module, torch.nn.Linear):
            prune.l1_unstructured(module, name='weight', amount=amount)
    return model

# --- Check sparsity ---
def sparsity(model):
    zeros = total = 0
    for p in model.parameters():
        zeros += (p == 0).sum().item()
        total += p.numel()
    return zeros / total * 100

# Apply pruning
model = prune_model(my_pretrained_model, amount=0.4)
print(f"Model sparsity: {sparsity(model):.1f}%")  # → ~40.0%

# Fine-tune briefly to recover accuracy
for batch in fine_tune_data:
    loss = model(batch)
    loss.backward()
    optimizer.step()

✅ Pros

Start from powerful base
Retains general knowledge

❌ Cons

Unstructured sparsity hard to accelerate
May need iterative pruning

2026-05-05 18:09

4
Quantization
Shrink numbers → shrink model
Reduce the precision of model weights from 32-bit floats down to 8-bit or 4-bit integers. A 7B model in 4-bit fits in 4 GB RAM.
Format Bits Size vs FP32 Quality
FP32 32 1× (baseline) Perfect
BF16 16 2× smaller Near- perfect
INT8 8 4× smaller Very Good
INT4(GPTQ) 4 8× smaller Good
2-bit 2 16× smaller Degraded

Simple Example
Python
— 4-bit quantization with bitsandbytes
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# Configure 4-bit quantization
quant_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_compute_dtype=torch.bfloat16,  # compute in BF16
  bnb_4bit_use_double_quant=True,     # nested quantization
  bnb_4bit_quant_type="nf4"        # NormalFloat4 format
)

# Load a 7B model in ~4 GB instead of ~28 GB
model = AutoModelForCausalLM.from_pretrained(
  "microsoft/phi-3-mini-4k-instruct",
  quantization_config=quant_config,
  device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-3-mini-4k-instruct")

# Run inference — now fits on a consumer laptop!
inputs = tokenizer("Explain gravity in simple terms:", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

2026-05-06 06:26
5
Fine-Tuning with LoRA / QLoRA
Domain-specialize a base SLM cheaply
Take a pretrained SLM (e.g., Llama 3.2-1B) and inject tiny trainable matrices into attention layers. Only ~0.1% of parameters are trained — the rest stay frozen.
How LoRA works
Instead of updating weight matrix W (millions of values), LoRA learns two tiny matrices A and B where A × B ≈ ΔW. This takes 1000× less memory.

Simple Example
Python
— QLoRA fine-tuning on domain data
from transformers import AutoModelForCausalLM, TrainingArguments, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
from datasets import load_dataset
import torch

# Step 1: Load base SLM in 4-bit (QLoRA)
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(
  "TinyLlama/TinyLlama-1.1B-Chat-v1.0", quantization_config=bnb
)

# Step 2: Attach LoRA adapters (only these will train)
lora_config = LoraConfig(
  r=16,             # rank of the low-rank matrices
  lora_alpha=32,         # scaling factor
  target_modules=["q_proj", "v_proj"], # which layers to adapt
  lora_dropout=0.05,
  task_type=TaskType.CAUSAL_LM
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # → ~0.1% trainable!

# Step 3: Fine-tune on your data (e.g., medical Q&A)
dataset = load_dataset("json", data_files="my_domain_data.jsonl")
trainer = SFTTrainer(
  model=model,
  train_dataset=dataset["train"],
  dataset_text_field="text",
  args=TrainingArguments(output_dir="./slm-finetuned",
              num_train_epochs=3, per_device_train_batch_size=4)
)
trainer.train()
model.save_pretrained("./my-domain-slm")

2026-05-06 06:28
6
Synthetic Data Training (Phi Approach)
Use GPT-4 to teach a tiny model
Use a large LLM to generate high-quality, "textbook-level" training examples, then train a small model on this curated data. This is exactly how Microsoft Phi-1, Phi-2, Phi-3 were built.

Design prompts for GPT-4
→
Generate synthetic corpus
→
Filter for quality
→
Train small model
→
Phi-3 level SLM

Simple Example
Python
— Generating synthetic training data via OpenAI
from openai import OpenAI
import json, random

client = OpenAI()

TOPICS = ["gravity", "photosynthesis", "recursion", "neural networks"]

def generate_sample(topic):
  """Ask GPT-4 to write a textbook explanation + question."""
  response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content":
      f"Write a clear, concise textbook paragraph about '{topic}' "
      "followed by 2 Q&A pairs. Format as JSON: "
      '{"text": "...", "qa": [{"q": "...", "a": "..."}]}'
    }]
  )
  return json.loads(response.choices[0].message.content)

# Generate dataset
dataset = []
for topic in TOPICS * 250:         # 1000 synthetic samples
  sample = generate_sample(topic)
  dataset.append(sample)
  print(f"Generated: {sample['text'][:60]}...")

# Save for training
with open("synthetic_corpus.jsonl", "w") as f:
  for item in dataset:
    f.write(json.dumps(item) + "\n")
print(f"Dataset ready: {len(dataset)} examples")
✅ Pros
Solves data quality problem
Works for any niche domain
Phi-3 beats models 10× its size
❌ Cons
Inherits teacher bias
GPT-4 API calls cost mon

2026-05-06 06:30
7
Efficient Architecture Design
Build smarter, not bigger
Use architectures designed from the ground up for small-scale efficiency: State Space Models, Mixture of Experts, or linear attention variants.

Simple Example — Mamba-style SSM block
Python
— Minimal State Space Model (SSM) block
import torch
import torch.nn as nn

class SimpleSSMBlock(nn.Module):
  """
  Simplified SSM block (Mamba-style concept).
  Replaces self-attention with a recurrent state update.
  Complexity: O(n) vs O(n²) for attention — great for long sequences.
  """
  def __init__(self, dim=128, state_dim=16):
    super().__init__()
    self.in_proj  = nn.Linear(dim, dim * 2)
    self.out_proj = nn.Linear(dim, dim)
    self.A = nn.Parameter(torch.randn(state_dim, dim)) # state matrix
    self.B = nn.Parameter(torch.randn(state_dim, dim)) # input matrix
    self.C = nn.Parameter(torch.randn(dim, state_dim)) # output matrix
    self.norm = nn.LayerNorm(dim)

  def forward(self, x):
    # x: (batch, seq_len, dim)
    B, T, D = x.shape
    h = torch.zeros(B, self.A.shape[0], device=x.device) # hidden state
    outputs = []

    for t in range(T):
      u = x[:, t, :]               # current token
      h = torch.tanh(h @ self.A.T + u @ self.B.T) # state update
      y = h @ self.C.T              # output projection
      outputs.append(y)

    out = torch.stack(outputs, dim=1)
    return self.norm(self.out_proj(out) + x)  # residual connection

# Test it
block = SimpleSSMBlock(dim=128)
x = torch.randn(2, 512, 128)  # batch=2, seq=512 tokens
out = block(x)
print(out.shape)  # → torch.Size([2, 512, 128])
✅ Pros
Linear complexity — faster on long text
No KV-cache growth
❌ Cons
Less mature ecosystem than transformers
Harder to parallelize training

2026-05-06 06:31

SLM Parameters

LLM Parameters (GPT-4)

SLM Inference Cost

LLM Inference Cost

7 Ways to Create an SLM

Simple Example

Pros

❌ Cons

Simple Example

✅ Pros

❌ Cons

Simple Example

✅ Pros

❌ Cons

Simple Example