🎓 A1Learner
🔍

PyTorch A Dataset from Files

A file-backed Dataset reads its rows from a file on disk, like a CSV table. The Dataset parses each text row into tensors once, when it is built.

A CSV row arrives as text

The csv module reads text. Every value arrives as a string, numbers included.

with open("shape_rows.csv", newline="") as f:
    reader = csv.reader(f)
    header = next(reader)
    first = next(reader)
print("Header:", header)
print("First row:", first)
print("Types:", [type(v).__name__ for v in first])

Parsing makes tensors

A file Dataset is a translator. Text goes in, and tensors come out.

features = torch.tensor([float(first[0]), float(first[1])])
label = int(first[2])
print("Features:", features)
print("Label:", label, type(label).__name__)

Batching the dataset

The file stays text on disk. The DataLoader stacks the parsed rows into tensor batches.

loader = DataLoader(dataset, batch_size=5)
features, labels = next(iter(loader))
print("Batch:", tuple(features.shape), "labels", labels.tolist())

Example

import csv

import torch
from torch.utils.data import Dataset, DataLoader

torch.manual_seed(0)

# The program writes its own small CSV first. No download is needed.
with open("shape_rows.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["filled", "mean", "label"])
    for i in range(12):
        label = i % 3
        filled = [40, 60, 64][label] + (i % 2)
        mean = round([0.16, 0.23, 0.25][label] + (i % 2) * 0.01, 2)
        writer.writerow([filled, mean, label])

print("Wrote shape_rows.csv with 12 rows")


class ShapeRows(Dataset):
    def __init__(self, path):
        self.rows = []
        with open(path, newline="") as f:
            reader = csv.reader(f)
            next(reader)  # skip the header row
            for filled, mean, label in reader:
                features = torch.tensor([float(filled), float(mean)])
                self.rows.append((features, int(label)))

    def __len__(self):
        return len(self.rows)

    def __getitem__(self, index):
        return self.rows[index]


dataset = ShapeRows("shape_rows.csv")
print("Rows loaded:", len(dataset))

loader = DataLoader(dataset, batch_size=5)
for features, labels in loader:
    print("Batch:", tuple(features.shape), "labels", labels.tolist())

Key points

  • CSV: a text table where commas separate the values in each row
  • Use float() and int() to turn CSV strings into numbers
  • Skip the header row before reading data rows
  • The closing program writes its own CSV first, so it needs no download

In this example, the program writes its own CSV first. The file shape_rows.csv gets a header row and 12 data rows. Each row holds a filled-pixel count, a mean value, and a label. The ShapeRows dataset then opens the file and skips the header. It parses each row into a features tensor and an int label, and stores the pairs. The dataset reports 12 rows. A DataLoader with batch_size 5 then serves three batches. Each batch prints its shape and its labels.

Note

csv is part of the Python standard library, so it needs no install.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

A CSV row reads ['40', '0.16', '0']. What must happen before training can use it?