🎓 A1Learner
🔍

PyTorch Activation Functions

An activation function is a function that reshapes a layer's numbers before the next layer. Without one, stacked linear layers act like a single linear layer.

ReLU keeps positives

ReLU sets negative numbers to zero. Positive numbers pass through unchanged.

values = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
print(torch.relu(values).tolist())

Sigmoid squeezes to 0 and 1

Sigmoid turns any number into a value between 0 and 1. Zero itself becomes exactly 0.5.

print([round(v, 3) for v in torch.sigmoid(values).tolist()])

Tanh squeezes to -1 and 1

Tanh turns any number into a value between -1 and 1. Negative numbers stay negative.

print([round(v, 3) for v in torch.tanh(values).tolist()])

Example

import torch
from torch import nn

values = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])

relu = nn.ReLU()
sigmoid = nn.Sigmoid()
tanh = nn.Tanh()

print("Input:  ", values.tolist())
print("ReLU:   ", [round(v, 3) for v in relu(values).tolist()])
print("Sigmoid:", [round(v, 3) for v in sigmoid(values).tolist()])
print("Tanh:   ", [round(v, 3) for v in tanh(values).tolist()])

Key points

  • ReLU: sets negatives to 0 and keeps positives as they are
  • Sigmoid: squeezes every number into the range 0 to 1
  • Tanh: squeezes every number into the range -1 to 1
  • Use torch.relu(x) to apply ReLU as a plain function
  • Use nn.ReLU() to place ReLU inside a model as a module
  • An activation reshapes numbers between layers, so the network can fit curves

In this example, one row holds five values from -2 to 2. The program builds the three activations as modules. Each module then runs on the same row. The prints line up as a small table. ReLU zeroes the two negatives and keeps the rest. Sigmoid squeezes every value between 0 and 1. Tanh squeezes every value between -1 and 1, so negatives stay negative. The values print rounded to three decimals so the table stays readable.

Note

Sigmoid and tanh flatten large inputs, so very big numbers land near the edge

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What does ReLU return for an input of -3?