I managed to write a really simple example of using threads in Python that I hope will give more insight on how to adapt my other programming stuff. And re-use this later on, in case I need to revisit this again it would be handy not to scour the internet again to assemble the bits and pieces of threading with Python.
The example below uses 3 threads, and processes 10 pairs of numbers (tuples) that I put in a list.
# Our list of work todo
inputlist_ori = [ (5,5),(10,4),(78,5),(87,2),(65,4),(10,10),(65,2),(88,95),(44,55),(33,3) ]
Those numbers are divided over those 3 threads by the Queue system.
The Queue system itself is limited to 5 slots, although this could easily be changed to more or less. You will notice in the console print that the message “Waiting for threads to finish.” appears after the fifth result, indicating that the queues are being used and the main program has continued on.
After putting everything in the queue system, the program waits for the threads to finish using the .join() function.
All spawned threads keep on being active, running forever, accepting jobs – that is, until the queue is empty, at which point they shut down.
I based most of my simple example on the examples in the Python threading tutorial (.pdf) work of Norman Matloff and Francis Hsu that I referenced before in a previous blog post. However, while their examples undoubtedly do more and are more extensive, they are also more complex. This example is deliberately made as simple as possible so to understand the basic principles of threading and the queue system.
Things I stumbled over:
- Duh! You spawn the threads before you fill up the queues with stuff todo…
- When printing out things to the console or python shell, things got jumbled because different threads took over from each other – to solve that I used the threading.Lock().acquire() and threading.Lock().release() to make sure that a thread could finish printing. Not sure if I understand completely all the possibilities this offers.
- Still a bit stumped on getting more info, name, etc on the thread that is running at the moment – haven’t figured that out yet how to do that.
Feel free to comment and ask questions – if you can improve this program, please let me know !
# threading test
# Alex Boschmans
# www.boschmans.net
# January 2010
#
# IMPORT SECTION
#
import threading, Queue
#
# Variables setup
#
THREAD_LIMIT = 3 # This is how many threads we want
jobs = Queue.Queue(5) # This sets up the queue object to use 5 slots
singlelock = threading.Lock() # This is a lock so threads don't print trough each other (and other reasons)
# Our list of work todo
inputlist_ori = [ (5,5),(10,4),(78,5),(87,2),(65,4),(10,10),(65,2),(88,95),(44,55),(33,3) ]
#
# This is called from the main function
# It spawns the threads, fills up the queue with work items that the threads will use
# And then waits for the threads to finish
# This could use some more try:except code...
#
def draadje(inputlist):
print "Inputlist received..."
print inputlist
# Spawn the threads
print "Spawning the {0} threads.".format(THREAD_LIMIT)
for x in xrange(THREAD_LIMIT):
print "Thread {0} started.".format(x)
# This is the thread class that we instantiate.
workerbee().start()
# Put stuff in queue
print "Putting stuff in queue"
for i in inputlist:
# Block if queue is full, and wait 5 seconds. After 5s raise Queue Full error.
try:
jobs.put(i, block=True, timeout=5)
except:
singlelock.acquire()
print "The queue is full !"
singlelock.release()
# Wait for the threads to finish
singlelock.acquire() # Acquire the lock so we can print
print "Waiting for threads to finish."
singlelock.release() # Release the lock
jobs.join() # This command waits for all threads to finish.
#
# Main thread class - based on threading.Thread
# This class is cloned/used as a thread template to spawn those threads.
# The class has a run function that gets a job out of the jobs queue
# And lets the queue object know when it has finished.
#
class workerbee(threading.Thread):
def run(self):
# run forever
while 1:
# Try and get a job out of the queue
try:
job = jobs.get(True,1)
singlelock.acquire() # Acquire the lock
print "Multiplication of {0} with {1} gives {2}".format(job[0],job[1],(job[0]*job[1]))
singlelock.release() # Release the lock
# Let the queue know the job is finished.
jobs.task_done()
except:
break # No more jobs in the queue
#
# Executes if the program is started normally, not if imported
#
if __name__ == '__main__':
# Call the mainfunction that sets up threading.
draadje(inputlist_ori)
Sigh. I just finished adding spaces to show where a def ends, and the damn code highlighter removed it again. Grrrrrr. If you want a copy of the code, let me know and I’ll update this post with a zipped copy of it.
Update: just discovered the “Syntaxhighlighter evolved” plugin and updated the code – indentation now works !!!
