Read Python List Item in Multiple Threads – Python Tutorial

Sometimes, we need to read each item in a python list to process. For example:

for i in data_list:
    #process(i)

However, this method is in a single thread. How to process each list item in multiple threads? In this tutorial, we will introduce you how to do.

How to create a thread in python?

It is to create a thread in python, here is the tutorial:

Create and Start a Python Thread with Examples: A Beginner Tutorial – Python Tutorial

How to read each item in a python list with multiple threads?

First, we can create a function to process each item in a python list. For example:

import threading
import time

def process(name, data_list):
    print("run thread "+ name)
    time.sleep(1)
    print(name+ " read data:", data_list)
    print(name+ " read data len:", len(data_list))
    print("thread " + name + " is finished")

Here data_list is a python list, we can process it in process() function. This function can be run in python thread.

Then, we can create a large python list and some threads.

all_data = [x for x in range(111)]
thread_num = 5

data_num_in_thread = len(all_data) // thread_num

all_thread = []

for i in range(thread_num):

    thread_name = 'thread{'+str(i)+'}'
    if i != (thread_num - 1):
        data_list = all_data[i*data_num_in_thread:(i+1)*data_num_in_thread]
    else:
        data_list = all_data[i*data_num_in_thread:]
    th = threading.Thread(target=process, args=[thread_name, data_list])
    all_thread.append(th)

In this example, we will create 5 threads to process a large python list all_data, which contains 111 items.

We can find each thread may process data_num_in_thread items in all_data.

Finally, we can start these threads as follows:

for th in all_thread:
    th.start()
for th in all_thread:
    th.join()
print("end!")

Run this example, we can see:

run thread thread{0}
run thread thread{1}
run thread thread{2}
run thread thread{3}
run thread thread{4}
thread{4} read data: [88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110]
thread{0} read data: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]
thread{3} read data: [66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87]
thread{2} read data: [44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65]
thread{1} read data: [22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]
thread{1} read data len: 22
thread thread{1} is finished
thread{2} read data len: 22
thread thread{2} is finished
thread{3} read data len: 22
thread thread{3} is finished
thread{4} read data len: 23
thread{0} read data len: 22
thread thread{4} is finished
thread thread{0} is finished
end!