Gilesthomas Domain Overview🔒 SSL Verified

Table of Contents

• ↓ Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090
• ↓ Putting my JAX-trained models on the Hugging Face Hub
• ↓ Why do OpenAI's GPT-2 weights beat mine? Part four: digging into dropout
• ↓ Adding diagrams to my static site generator with D2
• ↓ Use the built-in GELU, don't roll your own!
• ↓ A quick(ish) Chinchilla check
• ↓ Top Alternatives to gilesthomas.com

6-Month Traffic Trend Estimate

Official Contact & Profiles

Visual representation extracted from gilesthomas.com

Content Snippets from gilesthomas.com

"Mixture-of-experts models are really nifty. You get inference speed close to a small model's, with a lot of the smarts and knowledge of a large one. While they use as much memory as an equivalently-sized dense (non-MoE) model, they're much faster. The frontier labs don't publish their architectures, but Claude and ChatGPT are widely rumoured to be MoEs these days -- and certainly many large open-weights models like DeepSeek and Kimi K3 are."

"In this post, I'll show you how I added MoE support to the GPT-2-style code from Sebastian Raschka's book "Build a Large Language Model (from Scratch)", then used that to train a 446M-parameter model with 220M active parameters from scratch on my RTX 3090 -- essentially, GPT-2 small with 6 experts, 2 active per token. I wanted to understand how MoEs work, and as always, felt that the best way to do that is to build one (and then to write it up like this). Hopefully because it's all fresh in my mind as I write this, I should be able to explain things clearly for others who've finished Raschka's book."

"The training run took just less than eight days, and the resulting model got a better loss on my test set than any of the other models I've trained so far -- which was a good thing, given that it took four times longer to train! It was also better than the original OpenAI GPT-2 small (124M parameters), but not as good as GPT-2 medium (345M parameters), although it was close. That second result was interesting, as my model had more total parameters than GPT-2 medium, but fewer active ones."

"On an instruction fine-tuning test, it did better than any of my other models, but worse than both OpenAI models (why OpenAI's models are so good on that particular test is a mystery I'm digging into separately)."

Key Topics & Focus Areas for gilesthomas.com:

Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090

Putting my JAX-trained models on the Hugging Face Hub

Why do OpenAI's GPT-2 weights beat mine? Part four: digging into dropout

Adding diagrams to my static site generator with D2

Use the built-in GELU, don't roll your own!

A quick(ish) Chinchilla check