🎓 A1Learner
🔍

Python Choosing a Concurrency Tool

A concurrency tool is a way to run work overlapped or side by side.

Count words in order

The plain way runs each count in turn. It needs no extra tools.

def count_words(texts):
    total = 0
    for text in texts:
        total += len(text.split())
    return total

texts = ["the cat sat", "a dog ran fast"]
print(count_words(texts))

Count words with threads

A thread pool maps the same count over the texts. The structure adds a pool around the function.

from concurrent.futures import ThreadPoolExecutor

def count_one(text):
    return len(text.split())

texts = ["the cat sat", "a dog ran fast"]
with ThreadPoolExecutor(max_workers=2) as pool:
    results = list(pool.map(count_one, texts))
print(sum(results))

Count words with processes

A process pool has the same map shape. The work runs in separate processes instead.

from concurrent.futures import ProcessPoolExecutor

def count_one(text):
    return len(text.split())

def main():
    texts = ["the cat sat", "a dog ran fast"]
    with ProcessPoolExecutor(max_workers=2) as pool:
        results = list(pool.map(count_one, texts))
    print(sum(results))

if __name__ == "__main__":
    main()

Count words with asyncio

The async way wraps the count in a coroutine. gather() collects the counts in order.

import asyncio

async def count_one(text):
    await asyncio.sleep(0)
    return len(text.split())

async def main():
    texts = ["the cat sat", "a dog ran fast"]
    results = await asyncio.gather(
        *(count_one(text) for text in texts)
    )
    print(sum(results))

asyncio.run(main())

Example

import asyncio
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import ThreadPoolExecutor

def count_one(text):
    return len(text.split())

async def count_one_async(text):
    await asyncio.sleep(0)
    return len(text.split())

async def count_async(texts):
    results = await asyncio.gather(
        *(count_one_async(text) for text in texts)
    )
    return sum(results)

def main():
    texts = ["the cat sat", "a dog ran fast"]
    print(f"Sequential: {sum(count_one(t) for t in texts)}")
    with ThreadPoolExecutor(max_workers=2) as pool:
        print(f"Thread pool: {sum(pool.map(count_one, texts))}")
    with ProcessPoolExecutor(max_workers=2) as pool:
        print(f"Process pool: {sum(pool.map(count_one, texts))}")
    print(f"Asyncio: {asyncio.run(count_async(texts))}")

if __name__ == "__main__":
    main()

Key points

  • I/O-bound: work that spends its time waiting for input or output.
  • CPU-bound: work that spends its time calculating.
  • Use threads or asyncio for I/O-bound work.
  • Use processes for CPU-bound work, so cores can share the load.
  • Use asyncio when one program juggles many simple waits.
  • Use a thread pool when blocking calls already exist as normal functions.
  • Sockets: network connections.
  • This course only names them here.
  • Processes cost more to start than threads or tasks.
  • Concurrency tools run work overlapped or side by side.

In this example, one small job runs four ways. The job counts words in two short strings. The sequential line counts in a plain loop expression. The thread pool and process pool lines both use map(). Only the pool class changes between them. The asyncio line gathers async counts and sums them. Every line prints 7, so the tools differ in structure, not in the answer.

Note

Pick them when heavy CPU work earns that cost.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Your program spends most of its time on heavy maths. Which tool fits best?