raw-neural-net

This is a learning project and I would rather say so at the top than have you work it out. It follows the standard from-scratch curriculum, it is not an original architecture, and the repository is untidy in the way a repository is when its purpose was the writing rather than the result.

The goal was to stop treating the training loop as a black box. Calling model.fit() teaches you the API. It does not teach you what the gradient flowing back through a softmax actually looks like, and I wanted to know.

What is in it

Everything a small framework needs, written out: dense layers with L1 and L2 regularisation, dropout, ReLU, softmax, sigmoid and linear activations, four optimizers in SGD with momentum, Adagrad, RMSprop and Adam, four losses in categorical cross-entropy, binary cross-entropy, mean squared error and mean absolute error, and accuracy implementations for both classification and regression.

On top sits a model class that links layers to their neighbours, runs forward and backward passes, batches, validates, and saves and loads trained parameters.

NumPy does the array arithmetic. That is worth stating plainly, because “no framework” can be read as a bigger claim than it is: there is no autograd here, no computation graph, and no library computing a derivative for me. Every backward pass is written by hand. But the matrix multiplication is NumPy’s, and anyone opening the file sees that on line one.

The two places the understanding actually shows

Most of the code is mechanical once you have done the derivation. Two parts are not, and they are the reason the exercise was worth the time.

The fused softmax and cross-entropy backward pass. Taken separately, the derivative of softmax is a Jacobian per sample, and backpropagating through it is a matrix multiply you would rather not do. Composed with categorical cross entropy, almost all of it cancels, and the combined gradient collapses to something startling:

self.dinputs = dvalues.copy()
self.dinputs[range(samples), y_true] -= 1
self.dinputs = self.dinputs / samples

Subtract one from the predicted probability at the correct class, divide by the batch size, done. The first time you see that in a framework it looks like a trick. Deriving it yourself is what makes it obvious, and it is the single best argument for doing this exercise at all.

Adam’s bias correction. The momentum and cache terms both start at zero, so early in training they are biased toward zero and the first steps come out far too small. The correction divides each by one minus the decay rate raised to the step number, which is large early and decays to nothing:

weight_momentums_corrected = layer.weight_momentums / \
  (1 - self.beta_1 ** (self.iterations + 1))

The + 1 matters, and finding out why it matters is the kind of thing that only happens when you write it. The iteration counter starts at zero, and at zero the denominator would be exactly zero.

Then I photographed my own clothes

Fashion-MNIST is a solved dataset and a model scoring well on it has proved very little. So the real test was a shirt, a pair of trousers and a sneaker, photographed on my own phone.

That surfaced the thing the dataset hides. Fashion-MNIST is a light garment on a black background. A photograph of clothing is usually the opposite, a darker object against a lighter surface, so a model trained on one and shown the other is being handed a negative of what it learned. The preprocessing has to read the image as greyscale, resize to 28 by 28, and then invert it before scaling into the range the network was trained on.

Nothing about that is difficult and nothing about it is in the dataset documentation, because it is only visible once real input arrives. It is the smallest possible version of a lesson that shows up in every model I have deployed since: the training distribution is a choice somebody made, and reality did not agree to it.

It classified the photographs correctly. I did not record an accuracy figure at the time and I am not going to invent one now, so there is no number to quote here. Three garments is not an evaluation anyway. It was enough to answer the question I was asking, which was whether the thing worked outside the sandbox it was built in.

What this is worth

Not a portfolio piece in the sense of a system anyone runs. It is the answer to one interview question, asked in some form by everyone who has ever hired a machine learning engineer: does this person understand what the library is doing for them, or only how to call it.

I can tell you why softmax and cross entropy get fused, what Adam’s correction is for and why it disappears as training goes on, why dropout scales its output during training rather than at inference, and where regularisation enters the gradient. Not because I read it. Because the training loop did not converge until I got it right.