I mostly build experiments around one question: what actually happens inside a model when we train, damage, restore or transfer its capabilities?
Not everything here worked. Some experiments produced useful results, some produced negative ones. I keep both.
An experiment in recovering reasoning capabilities lost during fine-tuning.
After observing that conversational fine-tuning could degrade mathematical and general reasoning, I tested whether the damage could be localized to particular MLP layers rather than treated as a change to the entire model.
I used gradient-norm scoring to identify layers associated with errors, restored selected MLP weights from the original Qwen2.5-3B checkpoint, and evaluated the resulting models on GSM8K and ARC Challenge.
Restoring the top identified layers recovered much of the lost capability, with the strongest configuration reaching approximately the base model's ARC performance while slightly exceeding it on GSM8K.
An attempt to make knowledge distillation more selective.
Instead of asking a smaller student model to imitate its teacher uniformly, I compared hidden states between teacher and student layers and tried to concentrate the training signal where their internal representations diverged the most.
The first version catastrophically damaged the model. After protecting the final layers and adding an output-level KL anchor, the model recovered.
The localized method, however, still failed to outperform ordinary KL distillation. The result is negative — and I decided to keep it that way.
A tool for making internal MLP activity visible during language-model inference.
Given a prompt, Layer Vision captures MLP activations across transformer layers and shows which parts of the network respond most strongly.
The current version uses PyTorch forward hooks, allowing activations to be observed without modifying the model itself.
The project grew directly out of MLP Surgery. While measuring gradients and looking for damaged layers, I became interested in seeing how different prompts move through the model in the first place.
The long-term idea is to move from static layer measurements toward token-by-token visualization of activity inside the network.
An earlier experiment in dataset selection for language-model fine-tuning.
I scored OpenHermes samples using cross-entropy loss from a pretrained Qwen2.5-3B model and selected the highest-perplexity portion of the dataset.
The idea was simple: instead of spending compute on examples the model already understands, concentrate training on examples it finds difficult.
It didn't work.
After correcting an evaluation inconsistency, perplexity-filtered fine-tuning performed worse than the base model on both GSM8K and ARC Challenge.
That failure became useful. It exposed capability degradation during fine-tuning and eventually contributed to the idea behind MLP Surgery.
An ongoing experiment in model specialization.
Starting from Qwen3-4B-Instruct, the project explores separate specialized directions for mathematics, physics, chemistry, biology and code.
Still in development.
Nothing clever here.
Just Python.
A collection of small notebooks and exercises covering functions, classes, exceptions, files, data structures and other fundamentals.
I keep it public because not everything has to pretend to be research.