🎓 A1Learner
🔍

PyTorch The GAN Idea

A GAN pairs two models against each other, with a generator making fakes and a discriminator scoring them. Each model's loss depends on the other model, so both targets keep moving.

Noise becomes a fake value

The generator takes four noise numbers per row. It returns one fake value per row.

torch.manual_seed(11)
gen = nn.Sequential(nn.Linear(4, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
z = torch.randn(2, 4)
fake = gen(z)
print(fake.shape)
print(round(fake[0].item(), 2))

The discriminator scores single values

The discriminator takes one value and returns one score. A higher score means the value looks more real to it.

torch.manual_seed(11)
disc = nn.Sequential(nn.Linear(1, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
print(round(disc(torch.tensor([[3.0]])).item(), 4))
print(round(disc(torch.tensor([[0.0]])).item(), 4))

Right answers cost little

BCEWithLogitsLoss scores a discriminator output against its target. Confident scores aimed the right way cost little, and the same scores aimed the wrong way cost much more.

loss_fn = nn.BCEWithLogitsLoss()
scores = torch.tensor([2.0, -2.0])
print(round(loss_fn(scores, torch.tensor([1.0, 0.0])).item(), 4))
print(round(loss_fn(scores, torch.tensor([0.0, 1.0])).item(), 4))

Example

import torch
from torch import nn

torch.manual_seed(0)
real = torch.randn(8, 1) * 0.5 + 3.0


class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(4, 32), nn.ReLU(),
            nn.Linear(32, 32), nn.ReLU(),
            nn.Linear(32, 1))

    def forward(self, z):
        return self.net(z)


class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(1, 32), nn.ReLU(),
            nn.Linear(32, 32), nn.ReLU(),
            nn.Linear(32, 1))

    def forward(self, x):
        return self.net(x)


torch.manual_seed(11)
generator = Generator()
discriminator = Discriminator()
loss_fn = nn.BCEWithLogitsLoss()

z = torch.randn(8, 4)
fake = generator(z)

d_loss = (loss_fn(discriminator(real), torch.ones(8, 1))
          + loss_fn(discriminator(fake.detach()), torch.zeros(8, 1)))
g_loss = loss_fn(discriminator(fake), torch.ones(8, 1))

print("One real value:", round(real[0].item(), 2))
print("One fake value:", round(fake[0].item(), 2))
print("Discriminator loss:", round(d_loss.item(), 4))
print("Generator loss:", round(g_loss.item(), 4))

Key points

  • Generator: the model that turns noise into fake data
  • Discriminator: the model that scores values as real or fake
  • A good fake for the generator is a failure for the discriminator
  • Use detach on fakes during the discriminator step, so that step leaves the generator alone

In this example, eight real values come from a Normal spread with mean 3.0 and standard deviation 0.5. A seed then builds the generator and the discriminator. The generator turns eight noise rows into eight fakes. The first real value is 3.77, and the first fake is 0.08, far outside the real spread. The discriminator loss adds its score on the real values to its score on the detached fakes. It prints as 1.534. The generator loss asks the discriminator to call the fakes real, and it prints as 0.6774. Neither model takes a training step here. The next lesson runs the two steps in a loop.

Note

One round trains nothing here, so the printed losses are starting values only

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

In a GAN, what does the discriminator's loss reward?