🎓 A1Learner
🔍

Python ThreadPoolExecutor

A thread pool is a small team of worker threads ready to reuse. A task goes to the pool as one call to submit().

Threads managed by hand

Each thread here is created, started, and joined one by one.

import threading

results = [0, 0, 0]

def square(x, index):
    results[index] = x * x

threads = []
for i, n in enumerate([1, 2, 3]):
    t = threading.Thread(target=square, args=(n, i))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

print(results)

Submit one task

submit() hands the task to the pool. result() waits and gives back the return value.

from concurrent.futures import ThreadPoolExecutor

def square(x):
    return x * x

with ThreadPoolExecutor(max_workers=2) as pool:
    future = pool.submit(square, 5)
    print(future.result())

Map over many inputs

map() takes a function and many inputs. The results keep the input order.

from concurrent.futures import ThreadPoolExecutor

def square(x):
    return x * x

with ThreadPoolExecutor(max_workers=2) as pool:
    results = list(pool.map(square, [1, 2, 3, 4]))
print(results)

Handle futures as they finish

as_completed() yields futures in finish order. Sorting the values keeps this output stable.

from concurrent.futures import ThreadPoolExecutor, as_completed

def double(x):
    return x * 2

with ThreadPoolExecutor(max_workers=3) as pool:
    futures = [pool.submit(double, n) for n in [3, 1, 2]]
    values = [f.result() for f in as_completed(futures)]
print(sorted(values))

Errors surface at result()

A task can fail without stopping the pool. The error appears when you call result().

from concurrent.futures import ThreadPoolExecutor

def fail():
    raise ValueError("bad page")

with ThreadPoolExecutor(max_workers=1) as pool:
    future = pool.submit(fail)
    try:
        future.result()
    except ValueError as e:
        print(f"Error: {e}")

Example

import time
from concurrent.futures import ThreadPoolExecutor

pages = [
    ("Home", "Welcome to our site"),
    ("About", "We make small tools"),
    ("Guide", "Read this short guide first"),
    ("News", "Two new posts today"),
    ("Help", "Ask us anything"),
]

def fetch_length(page):
    title, text = page
    time.sleep(0.01)
    return len(text)

with ThreadPoolExecutor(max_workers=3) as pool:
    lengths = list(pool.map(fetch_length, pages))

for page, length in zip(pages, lengths):
    print(f"{page[0]}: {length}")
print(f"Total: {sum(lengths)}")

Key points

  • ThreadPoolExecutor: a pool of worker threads you can reuse
  • Use submit() to send one task and get a Future back
  • Future: a promise that a result will arrive later
  • Use result() to wait for the value and collect it
  • Use map() to run a function over many inputs in input order
  • Use as_completed() to handle each future as it finishes

In this example, pages holds five fake pages. Each page is a title and a short text. The fetch_length function pauses for a tiny moment. It then returns the length of the page text. The pool runs fetch_length over all five pages with map. map collects the lengths in the same order as the pages. The loop then pairs each page with its length and prints one line. The last line adds the lengths and prints 99.

Note

An exception in a task stays hidden until you call result()

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

You used pool.map() over five inputs. What order do the results come back in?