|
1 | 1 | # minitorch |
2 | 2 |
|
3 | | -This is a mini, torch-like deep learning library. The goal is that you can use syntax similar to PyTorch to build deep learning model using this library. This project starting points is the [minitorch exercises](https://github.com/minitorch/minitorch). However, after completing the exercises, I want to turn it into a functional deep learning library that can utilize GPU for model training, and I also want to refactor the code to make it clearer. This project is my attempt to do so. |
| 3 | +This is a mini, torch-like deep learning library. The goal is that you can use syntax similar to PyTorch to build deep learning model using this library. This project starting points is the [minitorch exercises](https://github.com/minitorch/minitorch). However, after completing the exercises, I want to turn it into a functional deep learning library that can utilize GPU for model training, and I also want to refactor the code to make it clearer. This project is my attempt to do so. |
| 4 | + |
| 5 | +## Installation |
| 6 | + |
| 7 | +```bash |
| 8 | +pip install -r requirements.txt |
| 9 | +``` |
| 10 | + |
| 11 | +## Usage |
| 12 | + |
| 13 | +### Basic Tensor Operations |
| 14 | + |
| 15 | +```python |
| 16 | +import minitorch |
| 17 | + |
| 18 | +# Create tensors with different backends |
| 19 | +FastTensorBackend = minitorch.TensorBackend(minitorch.FastOps) |
| 20 | +GPUBackend = minitorch.TensorBackend(minitorch.CudaOps) # If CUDA is available |
| 21 | + |
| 22 | +# Create a tensor |
| 23 | +x = minitorch.tensor([1, 2, 3, 4], backend=FastTensorBackend) |
| 24 | +y = minitorch.tensor([5, 6, 7, 8], backend=FastTensorBackend) |
| 25 | + |
| 26 | +# Perform operations |
| 27 | +z = x + y |
| 28 | +result = z.sum() |
| 29 | +``` |
| 30 | + |
| 31 | +### Building Neural Networks |
| 32 | + |
| 33 | +Create custom models by subclassing `minitorch.Module`: |
| 34 | + |
| 35 | +```python |
| 36 | +class Network(minitorch.Module): |
| 37 | + def __init__(self, backend): |
| 38 | + super().__init__() |
| 39 | + self.fc1 = minitorch.Linear(784, 128, backend=backend) |
| 40 | + self.fc2 = minitorch.Linear(128, 10, backend=backend) |
| 41 | + |
| 42 | + def forward(self, x): |
| 43 | + x = self.fc1(x).relu() |
| 44 | + x = minitorch.dropout(x, 0.2, not self.training) |
| 45 | + x = self.fc2(x) |
| 46 | + return minitorch.logsoftmax(x, dim=1) |
| 47 | +``` |
| 48 | + |
| 49 | +### Convolutional Neural Networks |
| 50 | + |
| 51 | +```python |
| 52 | +class CNN(minitorch.Module): |
| 53 | + def __init__(self, backend): |
| 54 | + super().__init__() |
| 55 | + self.conv1 = minitorch.Conv2d(in_channels=1, out_channels=6, kernel=(5, 5), stride=1, backend=backend) |
| 56 | + self.conv2 = minitorch.Conv2d(in_channels=6, out_channels=16, kernel=(5, 5), stride=1, backend=backend) |
| 57 | + self.fc1 = minitorch.Linear(16 * 4 * 4, 120, backend=backend) |
| 58 | + self.fc2 = minitorch.Linear(120, 10, backend=backend) |
| 59 | + |
| 60 | + def forward(self, x): |
| 61 | + batch_size = x.shape[0] |
| 62 | + x = self.conv1(x).relu() |
| 63 | + x = minitorch.avgpool2d(x, kernel=(2, 2), stride=(2, 2)) |
| 64 | + x = self.conv2(x).relu() |
| 65 | + x = minitorch.avgpool2d(x, kernel=(2, 2), stride=(2, 2)) |
| 66 | + x = x.view(batch_size, -1) |
| 67 | + x = self.fc1(x).relu() |
| 68 | + x = self.fc2(x) |
| 69 | + return x |
| 70 | +``` |
| 71 | + |
| 72 | +### Training Loop |
| 73 | + |
| 74 | +```python |
| 75 | +# Initialize model and optimizer |
| 76 | +model = Network(backend=FastTensorBackend) |
| 77 | +optimizer = minitorch.RMSProp(model.parameters(), lr=0.01) |
| 78 | + |
| 79 | +# Training |
| 80 | +model.train() |
| 81 | +for epoch in range(num_epochs): |
| 82 | + for X_batch, y_batch in train_loader: |
| 83 | + optimizer.zero_grad() |
| 84 | + output = model(X_batch) |
| 85 | + loss = minitorch.nll_loss(output, y_batch) |
| 86 | + loss.backward() |
| 87 | + optimizer.step() |
| 88 | + |
| 89 | +# Evaluation |
| 90 | +model.eval() |
| 91 | +for X_batch, y_batch in val_loader: |
| 92 | + output = model(X_batch) |
| 93 | + predictions = minitorch.argmax(output, dim=1) |
| 94 | +``` |
| 95 | + |
| 96 | +### Data Loading |
| 97 | + |
| 98 | +```python |
| 99 | +from minitorch.datasets import mnist |
| 100 | +from minitorch.dataloader import DataLoader |
| 101 | + |
| 102 | +# Load dataset |
| 103 | +mnist_train = mnist.MNISTDataset("/path/to/data", train=True) |
| 104 | + |
| 105 | +# Create dataloader |
| 106 | +train_loader = DataLoader( |
| 107 | + mnist_train, |
| 108 | + batch_size=32, |
| 109 | + shuffle=True, |
| 110 | + backend=FastTensorBackend, |
| 111 | + transform=lambda x: x.astype(np.float64) / 255.0 |
| 112 | +) |
| 113 | +``` |
| 114 | + |
| 115 | +### GPU Acceleration |
| 116 | + |
| 117 | +```python |
| 118 | +import numba |
| 119 | + |
| 120 | +# Check CUDA availability and use GPU backend |
| 121 | +if numba.cuda.is_available(): |
| 122 | + backend = minitorch.TensorBackend(minitorch.CudaOps) |
| 123 | + print("Using GPU backend") |
| 124 | +else: |
| 125 | + backend = minitorch.TensorBackend(minitorch.FastOps) |
| 126 | + print("Using CPU backend") |
| 127 | + |
| 128 | +model = Network(backend=backend) |
| 129 | +``` |
| 130 | + |
| 131 | +### Saving and Loading Models |
| 132 | + |
| 133 | +```python |
| 134 | +# Save model weights |
| 135 | +model.save_weights("model.npz") |
| 136 | + |
| 137 | +# Load model weights |
| 138 | +model.load_weights("model.npz") |
| 139 | +``` |
| 140 | + |
| 141 | +### Available Optimizers |
| 142 | + |
| 143 | +- `SGD(parameters, lr, momentum)` - Stochastic Gradient Descent with optional momentum |
| 144 | +- `RMSProp(parameters, lr, decay_rate, eps)` - RMSProp optimizer |
| 145 | + |
| 146 | +### Available Loss Functions |
| 147 | + |
| 148 | +- `nll_loss(output, target)` - Negative Log Likelihood Loss |
| 149 | +- `bce_loss(output, target)` - Binary Cross Entropy Loss |
| 150 | +- `cross_entropy_loss(output, target)` - Cross Entropy Loss |
| 151 | +- `mse_loss(output, target)` - Mean Squared Error Loss |
| 152 | + |
| 153 | +### Example: Training MNIST |
| 154 | + |
| 155 | +See `examples/run_mnist_multiclass.py` for a complete example of training a LeNet-5 CNN on MNIST: |
| 156 | + |
| 157 | +```bash |
| 158 | +python examples/run_mnist_multiclass.py --backend gpu --batch_size 32 --epochs 10 --lr 0.01 |
| 159 | +``` |
0 commit comments