🎓 A1Learner
🔍

Python Queues Between Threads

A queue is a safe line where one thread hands values to another. One thread puts jobs into the queue, and another takes them out and does the work.

Put and get

Items come out in the order they went in. This is a first-in, first-out line.

import queue

jobs = queue.Queue()
jobs.put("wash")
jobs.put("dry")
print(jobs.get())
print(jobs.get())

Mark a job done

Each get() needs one matching task_done(). Then join() can return.

import queue

jobs = queue.Queue()
jobs.put("email")
item = jobs.get()
print("Working on:", item)
jobs.task_done()
jobs.join()
print("Queue is empty and done")

A sentinel stops the consumer

Here None is the sentinel. The consumer stops when it gets None.

import queue
import threading

jobs = queue.Queue()
done = []

def consumer():
    while True:
        item = jobs.get()
        if item is None:
            jobs.task_done()
            break
        done.append(item.upper())
        jobs.task_done()

t = threading.Thread(target=consumer)
t.start()
jobs.put("pen")
jobs.put("cup")
jobs.put(None)
jobs.join()
t.join()
print(sorted(done))

Other queue types

LifoQueue takes the newest item first. SimpleQueue is a plain queue with fewer features.

import queue

stack = queue.LifoQueue()
plain = queue.SimpleQueue()

Example

import queue
import threading

def producer(jobs):
    for number in [9, 4, 15, 2, 11]:
        jobs.put(number)
    jobs.put(None)

def consumer(jobs, results):
    while True:
        number = jobs.get()
        if number is None:
            jobs.task_done()
            break
        results.append(number * 2)
        jobs.task_done()

if __name__ == "__main__":
    jobs = queue.Queue()
    results = []
    worker = threading.Thread(target=consumer, args=(jobs, results))
    worker.start()
    producer(jobs)
    jobs.join()
    worker.join()
    print("Completed:", sorted(results))
    print("All jobs done")

Key points

  • Queue: a safe line where one thread hands values to another.
  • Use put() to add an item to a queue.Queue.
  • Use get() to take the next item from the queue.
  • Use task_done() after you finish one item from get().
  • Use join() on the queue to wait until every item is done.
  • Producer: the side that puts jobs into the queue.
  • Consumer: the side that gets jobs and does the work.
  • Sentinel: a special value that tells a consumer to stop.
  • Use SimpleQueue for a plain queue with no task tracking.
  • Use LifoQueue when the newest item should go first.
  • SimpleQueue has no task_done() or join() tracking.

In this example, producer puts five numbers into the queue. It then puts None as the sentinel. The consumer thread takes one number at a time. It doubles each number and saves the result. It calls task_done() after each item. It stops when it gets the sentinel. The main code waits with jobs.join() and then worker.join(). It prints the results sorted, so the report order is fixed.

Note

Use Queue when you must wait for finished work.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What does the sentinel value do in a producer and consumer queue?