Using threads in Python

I’ve been trying to setup threading in Python, so that in the back-end of my service system that I’m developing I can query more than one source at the same time. So instead of querying one server and waiting for feedback, I can launch 10 threads and thus query 10 servers and process each server’s feedback via it’s own thread.

So a very vague, generalising definition of a thread is an independent ‘process’ that performs a job that you give it. You can control how many threads that you launch. Each thread is a copy of the original thread that you describe (in essence a python def function that has been wrapped in a thread class).

Right now, my understanding of threads is a bit confused. So far it seems that threading has several different manners of implementing them:

  • using a number of threads that you launch, use, and forget about them (they go away)
  • improving on that by putting those threads in a thread pool, and when a thread finishes, re-using it for the next job (so you haveĀ  5 threads but 10 jobs to do, those five threads take five jobs, and the first thread that finishes takes on the sixth job, the second thread to finish the seventh job, and so on)
  • the final step seems to be (I haven’t got that far in my implementation) to set up worker bees that are managed by one thread (a better description is promised, as soon as I have understood it!)

Since I’ve been scouring the net for information over threads, here is a list of resources that discuss, give examples, and explain threads – it’s useful for me to refer to, it might be useful for you as well :

  • DaveN has an extensive post, with examples, building up gradually. It’s only at the end that you read that the code shown has never been run, which is a bit of a letdown. Still worth a good read though !!
  • A very thorough 25-page pdf documents that starts from the beginning is available on the site of UC Davis, University of California. It goes into all the nitty gritty details.
  • An example that uses workers in threads is found on the blog of Danial Taherzadeh.
  • Another one that discusses using multiple queues chained together can be found on IBM’s developerworks site.
  • And the blog post from Halotis that started my looking into threads…

Right now I’m using threads in a thread pool, but I’m not doing something right – I noticed that while I have 10 jobs to do, only the first five get done, and the others ‘disappear’.

I guess the only way to get it working is to continue reading the information above until it makes sense. Sometimes I wonder if I’m not slightly masochistic, looking for challenges like that… ai me poor pounding head ! :-)

One thought on “Using threads in Python”

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>