Python threading + deque to Process List Data – Python Tutorial

By | November 3, 2023
0 Comment

Sometimes, we need to process a large of list data, we usually want to create some threads to process. In this tutorial, we will introduce you how to do.

If you only want to use python threading, you can view this tutorial:

Read Python List Item in Multiple Threads – Python Tutorial

However, in this tutorial, we will introduce a simpler method: threading + deque

Preliminary

We should import some libraries.

from collections import deque
import threading
import time

Create some python threads and a deque to process list

Here is an example:

def process_list(data_list, thread_num = 5, fun = None):
    d = deque(data_list)

    all_thread = []
    for i in range(thread_num):
        thread_name = 'thread{' + str(i) + '}'
        th = threading.Thread(target=fun, args=[thread_name, d])
        all_thread.append(th)

    for th in all_thread:
        th.start()
    for th in all_thread:
        th.join()
    print("end!")

In this code, we use deque(data_list) to crease a deque d using a python list.

Then, we will use fun to process deque d.

This fun is defined as:

def process(name, d):
    print("run thread "+ name)
    while d:
        #time.sleep(1)
        item = d.popleft()
        print(name+ " read data:", item)
    print("thread " + name + " is finished")

In order to check a deque is empty or not, we can use if d or while d.

Finally, we can use these code as follows:

process_list(data_list, 5, process)

Run this code, we will see:

Python threading + deque to Process List Data - Python Tutorial