Skip to main content

How well do you know python, part 8

Here is a mostly-functioning class that facilitates writing producer / consumer algorithms. It is designed to support a single producing and multiple consuming threads, i.e., scenarios where consuming involves some blocking operations such as communicating over a socket.

"Mostly," in this case, means that there are two bugs in this class that can be fixed in a total of four or five lines. One is arguably more subtle than the other, but they involve the same part of the code. Can you spot them?

Warning: this one is tougher than part 2, which also dealt with threading.

import Queue as queue
import threading

class PCQueue:
    def __init__(self, initialthreads):
        self.q = queue.Queue()
        self.running = True
        self.condition = threading.Condition()
        for i in range(initialthreads):
            self.add_consumer()
    def run_consumer(self):
        while True:
            self.condition.acquire()
            try:
                self.condition.wait()
                if not self.running:
                    return
            finally:
                self.condition.release()
            obj = self.q.get()
            self.consume(obj)
    def put(self, obj):
        self.condition.acquire()
        self.q.put(obj)
        self.condition.notify()
        self.condition.release()
    def consume(self, obj):
        raise 'must implement consume method'
    def add_consumer(self):
        threading.Thread(target=self.run_consumer).start()
    def stop(self):
        self.running = False
        self.condition.acquire()
        self.condition.notifyAll()
        self.condition.release()

Here's a short example using PCQueue.

class PCExample(PCQueue):
    def consume(self, url):
        import urllib
        n = len(urllib.urlopen(url).read())
        print '\t%s contains %d bytes' % (url, n)

q = PCExample(2) # 2 threads
while True:
    print 'url? (empty string to quit)'
    url = raw_input()
    if not url:
        break
    q.put(url)
q.stop()

Comments

the paul said…
Well, I haven't run this, so I don't know if there's something even more obvious that I'm missing, but..

(a) It looks like it's possible for the put() method to notify the threads at a moment when none of them are actually waiting. The item would get stuck in the queue, probably forever. Trying to call self.q.get_nowait() in run_consumer before the wait would probably fix this.

(b) The notifyAll() call in stop could also happen when some of the threads aren't waiting, so they would wait for another notification before checking self.running. Similarly to the last item, putting another check for self.running before the wait would solve this.

(b) the use of the condition.notify() in put() isn't quite future-proof; the docs say future versions of notify might sometimes wake up more than one thread, and the code doesn't allow for that. This is probably not an issue.

I have a feeling I didn't catch the subtle bug.
the paul said…
Er, on (a), I meant that some item would always be stuck in the queue, not that that specific item would remain stuck.
Anonymous said…
I don't know Python this well, but your challenges are interesting.

I tried the program and it seems to run well, but anyway :

In run_consumer there is:
self.condition.wait()
if not self.running:
return
shouldn't the condition be released before the return?
Jonathan Ellis said…
Paul, those are the issues I had in mind.

Damjan: that is why that code is in a try/finally block. The code in a finally block always gets executed after the try block completes, even if the try block terminated because of a break or return.
Anonymous said…
jonathan: on the first gleam I saw the finally: statement as an except: .

Then I added a print "Condition released" just after the finally: statement to make sure the condition is released.
the paul said…
This one was easier to me than the part 2 question, for which I had to decode Ian Bicking's answer. I suck!

Popular posts from this blog

PyCon Python IDE review

I presented an IDE review at PyCon last Friday. It was basically a re-review of what I thought were the 3 most promising IDEs from the Utah Python User Group IDE review , to which I added SPE, which was by far the most popular of the ones we left out that time. The versions reviewed are: PyDev 1.0.2 SPE 0.8.2.a Komodo 3.5.2 Wing IDE 2.1 beta 1 I'd intended to base my presentation around a comparison of writing a smallish program in each of the IDEs, but the more I tried to make this not suck, the more I realized it was a losing proposition. Instead, I decided to try to focus on the features in each that most set them apart from the others (both positive and negative); this seemed more likely be useful. (I did a new feature matrix for this review, which is included after my comments. The slides I used are also up, at http://utahpython.org/jellis/pycon-ides.pdf , but aren't very useful absent video of the presentation itself. Hence this post.) PyDev PyDev has g...

Why PHP sucks

(July 8 2005) Apparently I got linked by some PHP sites, and while there were a few well-reasoned comments here I mostly just got people who only knew PHP reacting like I told them their firstborn was ugly. These people tended to give variants on one or more themes: All environments have warts, so PHP is no worse than anything else in this respect I can work around PHP's problems, ergo they are not really problems You aren't experienced enough in PHP to judge it yet As to the first, it is true that PHP is not alone in having warts. However, the lack of qualitative difference does not mean that the quantitative difference is insignificant. Similarly, problems can be worked around, but languages/environments designed by people with more foresight and, to put it bluntly, clue, simply don't make the kind of really boneheaded architecture mistakes that you can't help but run into on a daily baisis in PHP. Finally, as I noted in my original introduction, with PHP, ...

A review of 6 Python IDEs

(March 2006: you may also be interested the updated review I did for PyCon -- http://spyced.blogspot.com/2006/02/pycon-python-ide-review.html .) For September's meeting, the Utah Python User Group hosted an IDE shootout. 5 presenters reviewed 6 IDEs: PyDev 0.9.8.1 Eric3 3.7.1 Boa Constructor 0.4.4 BlackAdder 1.1 Komodo 3.1 Wing IDE 2.0.3 (The windows version was tested for all but Eric3, which was tested on Linux. Eric3 is based on Qt, which basically means you can't run it on Windows unless you've shelled out $$$ for a commerical Qt license, since there is no GPL version of Qt for Windows. Yes, there's Qt Free , but that's not exactly production-ready software.) Perhaps the most notable IDEs not included are SPE and DrPython. Alas, nobody had time to review these, but if you're looking for a free IDE perhaps you should include these in your search, because PyDev was the only one of the 3 free ones that we'd consider using. And if you aren...