Neural Networks from Scratch

This repository is personal learning: I implemented the main pieces of a neural network from scratch in NumPy. The goal was to understand the calculations inside each piece by writing them, rather than to call a high-level deep-learning library. The themes follow volume 1 of 斎藤康毅’s 『ゼロから作るDeep Learning ― Pythonで学ぶディープラーニングの理論と実装』, and I did not use the book’s code or the official repository.

NumPy handles the numerical substrate: storing arrays, matrix products, element-wise operations, reductions, reshape, and random numbers. Layers, losses, optimizers, and the training loop are my own. Frameworks such as PyTorch are not used to implement or train the network itself.

Layers

I implemented Linear, ReLU, Sigmoid, Flatten, Conv2D, MaxPooling2D, Dropout, BatchNormalization, and Sequential. Each layer has a forward and a backward pass. Conv2D is a two-dimensional convolution with stride and padding. MaxPooling2D selects the maximum in each window and backpropagates through that choice. Sequential stacks layers into an MLP or a CNN.

Loss and optimization

The losses are MeanSquaredError, CrossEntropy, and BCEWithLogits. The forward pass computes the loss, and the backward pass computes the gradient with respect to the input. Where the arithmetic is numerically fragile, the implementation stays stable where it can.

The optimizers are SGD, Momentum, AdaGrad, and Adam. Adam includes the first moment, the second moment, and bias correction. DataLoader builds mini-batches, and Trainer keeps the loop that ties the model, loss, optimizer, and DataLoader together, separate from the network itself.

Tests

Accuracy on a dataset is not the only check. Tests confirm that the forward pass matches the expected value on known inputs, that backward gradients match for the input, the weights, and the bias, and that shapes hold through both passes. Convolution and pooling also cover rectangular kernels, different strides and padding, overlapping regions, and multi-channel inputs. Simple inputs are compared with hand calculation or an independent reference. PyTorch and similar libraries are used only as a numerical reference for the implementation, not as the implementation.

I used AI to design and review test cases for verification. The network itself is implemented after understanding the algorithm.

Checking that training runs

Comparing optimizers and training at large scale were out of scope. On small data, an MLP and a CNN with enough capacity overfit the training set, which shows that loading data, computing the loss, backpropagating, and updating parameters work as one pipeline. The components, the tests, these two checks, and a passing automated test run are done.

The implementation and tests are on GitHub.

Stack

• Language: Python
• Numerics: NumPy
• Environment: uv