PyTorch Your First Network

A first network is a small stack of layers that turns input numbers into scores. This one is an MLP, a stack of linear layers with activations between them.

The whole module

The class builds three parts in __init__. forward runs them in order, hidden first and output last.

class FirstNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(3, 8)
        self.relu = nn.ReLU()
        self.output = nn.Linear(8, 2)

    def forward(self, x):
        x = self.hidden(x)
        x = self.relu(x)
        return self.output(x)

Shapes through the network

One point grows to eight hidden numbers. It then shrinks to two scores.

torch.manual_seed(0)
model = FirstNet()
one = torch.tensor([[1.0, 0.0, 0.5]])
print(model.hidden(one).shape)
print(model(one).shape)

Raw scores for one point

The scores are raw numbers from the last layer. They are not probabilities, and one of them is negative.

with torch.no_grad():
    scores = model(one)
print([round(v, 3) for v in scores[0].tolist()])

Example

import torch
from torch import nn

torch.manual_seed(0)


class FirstNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(3, 8)
        self.relu = nn.ReLU()
        self.output = nn.Linear(8, 2)

    def forward(self, x):
        x = self.hidden(x)
        x = self.relu(x)
        return self.output(x)


model = FirstNet()
points = torch.tensor([[1.0, 0.0, 0.5],
                       [0.0, 1.0, -0.5],
                       [0.5, 0.5, 1.0],
                       [-1.0, 0.0, 0.0]])

with torch.no_grad():
    scores = model(points)

print("Input shape:", points.shape)
print("Scores shape:", scores.shape)
for row in scores.tolist():
    print("Scores:", [round(v, 3) for v in row])

Key points

MLP: short for multi-layer perceptron, a stack of linear layers with activations
Hidden layer: a layer between the input layer and the output layer
Score: a raw number from the last layer, before any decision is made
Use torch.no_grad() for a forward pass that needs no gradients
A network is modules in a row, and a forward pass runs input through the row

In this example, the seed is set first, so the starting weights are repeatable. The FirstNet class builds its parts in __init__. The hidden layer makes eight numbers from three inputs. ReLU sets any negative numbers to zero. The output layer makes two scores from the eight numbers. forward runs the three parts in that order. The points tensor holds four made-up rows of three numbers. The forward pass runs under no_grad, since no training happens here. The prints show both shapes, then one scores row per point. The scores print rounded to three decimals. Training this network comes later, after losses and optimizers.

Note

The scores are not probabilities, and they do not have to sum to 1

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

import torch
from torch import nn

torch.manual_seed(0)


class FirstNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(3, 8)
        self.relu = nn.ReLU()
        self.output = nn.Linear(8, 2)

    def forward(self, x):
        x = self.hidden(x)
        x = self.relu(x)
        return self.output(x)


model = FirstNet()
points = torch.tensor([[1.0, 0.0, 0.5],
                       [0.0, 1.0, -0.5],
                       [0.5, 0.5, 1.0],
                       [-1.0, 0.0, 0.0]])

with torch.no_grad():
    scores = model(points)

print("Input shape:", points.shape)
print("Scores shape:", scores.shape)
for row in scores.tolist():
    print("Scores:", [round(v, 3) for v in row])

Exercise

How many scores does this network return for each input point?

8 3 2 4