Tuesday, December 06, 2005

Spyce on Wikipedia

Spyce user Lars Schmidt created a Wikipedia article on Spyce. Nice!

(Now I need to add a section on Active Handlers, which is one of the main things I think makes Spyce compelling for larger projects, as well as simplifying smaller ones.)

Wednesday, November 30, 2005

A point for ORM developers to remember

If you are writing an ORM tool, please keep this one point in mind:

I already know SQL.

Your new query syntax isn't better; it's just different. Which means one more thing for me to learn. Which means I probably won't bother, and will use instead an ORM tool that respects my time.

Consider: if your target audience is like me, and already knows SQL, this should go without saying. Resist the temptation to be "clever" and overcomplicate things.

If your target audience does not know SQL, then either

  • they intend to learn SQL, because they're using a relational database and expect that non-python code or ad-hoc queries will be necessary at some point, or
  • they have no intention of learning SQL, and all their data-access code will now and forever be in python, in which case they would be better off using Durus or ZODB or another OODB. Be honest with these people and everyone will be happier.

Django does some cool stuff, but heaven help me if I ever had to use an abomination like their ORM in its present form. It's not pretty. Look, here's a rule of thumb to tell when your code sucks: if it reminds the reader of perl, it sucks. There. The secret of not sucking is yours. Repent and suck no more!

At the risk of proceeding to beat a dead horse, didn't anyone look at those code samples and think, "wow, our ORM code is way the hell uglier than the vanilla SQL?"

But even if you discover the World's Prettiest And Most Functional Syntax, resist the temptation to make me use it. Remember: I already know SQL. (Corollary: don't bother giving SQL functionality a facelift, as in "startswith='WHO'" instead of "LIKE 'WHO%'.")

I only single out Django here because they blogged about how cool their ORM syntax for ORs was, which made me have a look, which prompted this rant. Sorry, Django fans!

(Java is clunky and ugly but the Java SimpleORM tool is better thought-out than anything I have seen in Python, and does not make this mistake. Read the author's whitepaper.)

Tuesday, November 29, 2005

PATA really, really sucks

Trying to figure out why sometimes disk access on my test machine takes way, way too long -- 1000+ ms -- I wrote some test code. My threads ran a function that looks like this:

write = []
def writer():
    while True:
        start = time.time()
        f = tempfile.TemporaryFile()
        f.write('a' * 4000)
        end = time.time()
        write.append(end - start)

Compare the times for max(write) on a machine with a SATA disk and on one with parallel ATA, where the given number of threads are run for a 10 second period:

threads         pata    sata
1               6ms      6ms
2               400ms   11ms   
4               1300ms  24ms

Ouch.

I admit I'm not a hardware nerd. Quite possibly I'm missing something, because even PATA shouldn't be THAT bad. Right? hdparm -i says for the PATA disk:

/dev/hda:

 Model=ST380011A, FwRev=8.01, SerialNo=4JV59KZT
 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
 BuffType=unknown, BuffSize=2048kB, MaxMultSect=16, MultSect=off
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=156301488
 IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: ATA/ATAPI-6 T13 1410D revision 2:

 * signifies the current active mode

Tuesday, October 18, 2005

Startup school

I attended Paul Graham's Startup School this past Saturday. (Thanks to Drew Houston for letting me crash at his place!) I took fairly detailed notes on the speakers; I think this is the most comprehensive overview available, actually. (Note that these are in the order they actually spoke, which isn't quite the same as the plan.)

  1. Langley Steinert, entrepreneur. Short "entrepreneurship 101" talk. Lots of interesting Q&A.
  2. Marc Hedlund, Entrepreneur in Residence, O'Reilly Media. Talks about over a dozen startups he's seen in the last couple years -- most of which you'll recognize -- and why they succeeded.
  3. Qi Lu, VC at Yahoo. One subject: Why yahoo rocks. Total advertorial; skip this one unless you're a yahoo fanboy.
  4. Hutch Fishman, part-time CFO for startups. Talks about VC founds, directors, liquidity. Pretty basic stuff if you've read up on this at all, but his Q&A is worth reading.
  5. Paul Graham, author of entrepreneurship essays, VC, and Lisp fan. This was my first time seeing Paul speak; I was surprised to find out that when one of his essays says "... derived from a talk at ..." it really means "I read this essay at ..." So you might as well read the transcript instead of my notes, up until the Questions section. (Respect to Reg Braithwaite, who stood up to challenge the conventional wisdom that entrepreneurship isn't for people with kids.)
  6. David Cavanaugh, lawyer. Standard slashdot-style IP overview. If you know what the differences are between patent, copyright, trademark, and trade secret, you won't see anything new here.
  7. Michael Mandel, economist. Before the lunch break, I told my friends that this talk was either going to be very interesting or very dull. I'm glad I was right (and it wasn't dull). One of his points is, "The USA has certain systemic advantages that encourage entrepreneurship and growth." But I hesitate to call that his main point, since he talks about a lot more than that. Worth reading.
  8. Steve Wozniak. Sort of a retrospective. I don't know that I learned much, but hey, I got to shake Woz's hand.
  9. Mark Macenka, IP lawyer. Somewhat more useful than the other lawyer's talk. Read the "Founders issues" section.
  10. Stan Reiss, VC. Interesting take on startups from a VC's perspective: when taking VC money is worth it, and when you shouldn't. Long Q&A session.
  11. Stephen Wolfram, founder of the Mathematica company. I figured when I saw shots of cellular automata on the projector while he set up that this wouldn't be about entrepreneurship, and I could tune it out. I was right.
  12. Chris Sacca, head of new business development for Google. As much of a cheerleader as Qi Lu was, but less blatant and more interesting.
  13. Olin Shivers, professor and entrepreneur. Talks about failure and attitude in a Nietsche-ish way. Moved really, really fast; Olin was the only speaker I felt like I couldn't keep up with, note-taking-wise. Catch the video when it goes up. (If you just look at his slides, you'll miss a lot.) Excellent talk.
  14. Summer founders, the guys (all of them were male) who took seed money from Paul's VC firm. Probably most interesting if you're college-age-ish and also thinking about applying to Y Combinator.

See also the slides for those presenters who had them. Supposedly videos will be up eventually.

Of the other blogs I've seen, Fred Ngo's and Robbie Allen have the best speaker coverage.

Thanks to Paul and the speakers for organizing this! I had fun and learned a lot.

Sunday, October 16, 2005

Why I never got into Lisp

Like many programmers (I suspect) who hear enough Lisp afficionados talk about how great their language is, I've given Lisp a try. Twice. I didn't make much headway on either occasion.

For various reasons I was poking around python/lisp in google tonight and I came across this thread from a couple years ago. This crystalized it for me:

now compare
        if x > y or a == 0:
            a1 = m * x + b
        else:
            a2 = m / x - b

with
        (if (or (> x y ) (zerop a))
            (setq a1 (+ (* m x ) b )
            (setq a2 (- (/ m x ) b )).

In this case, Lisp requires about 33% more typing, almost all parentheses.

And, oh yeah, I forgot, aside from the parentheses, the pervasive prefix notation can be a drag.

Syntax does matter. Everyone who has run screaming from a steaming pile of Perl understands this. Lisp's syntax isn't the train wreck that Perl's is; at the very least it's consistent. I could get used to it if there were no alternatives.

But Lisp syntax was designed 50 years ago to make it easy to parse! We don't have to optimize for this anymore -- it's time to optimize for the programmer!

Today, Python and others have the features that made Lisp powerful without the syntactical baggage. Python has its own advantages, too, such as one of the best-designed standard libraries around. (Hence Paul Cannon's Lisp-to-python-bytecode compiler.)

Further reading: On the Relationship Between Python and Lisp.

Thursday, October 13, 2005

Why do Java programmers like Ruby?

As a (mostly ex-) Java programmer myself who prefers Python to Ruby, I'm puzzled by what seems like a rush of Java programmers to embrace Ruby as though it were the only dynamic language on the planet.

I understand that it's mostly because of the success of Rails, which definitely came at the right time with the right marketing. But Ruby really doesn't seem like a good philosophical match with Java to me.

Java, to a large degree, tried to be "C++ done right." That is, C++ without all the misfeatures that seemed good at the time but whose benefit turned out to not be worth the cost in complexity for developers. Java is a very orthogonally designed language; there is usually one obvious way to accomplish something. Python shares this. Ruby, OTOH, takes the C++ and Perl philosophy of "there's more than one way to do it," with predictible effects on maintainability. ("Perl," said a former co-worker to me yesterday, "is the drunken frat boy of languages.")

I could point at other areas where Python seemed like a better fit to me, like real threading support, etc., but the core issue seems to be: what is the language's philosophy? Is it trying to help my team write maintainable, readable code, or is it more interested in being "clever?"

Java and Python were designed for readability. C++ and Ruby were designed to be clever.

About the only thing Ruby has going for it, from this Java developer's perspective, is braces. Ahh, comforting braces. Are people that afraid of syntactical change?

I guess now I know how Lisp developers feel, a little.

Tuesday, October 11, 2005

Microsoft invents significant whitespace

Brace you must
     Written it is, the Console. “Hello World”

YODA the programming language

Wednesday, October 05, 2005

John Siracusa on C# and Java

John, of course, writes the OS X reviews for ars technica. (If you haven't read these, you need to. His level of detail should make lesser reviewers think hard about finding another career.)

In his blog, he's been talking about what Apple needs to do to avoid another Copland scenario, i.e., avoid resting on their OS X achievments only to wake up in 2010 and realize, "Oh crap! It's WAY easier to write apps for Windows than for OS X!"

Which is a rather long introduction to the quote I liked so much:

My instincts tell me that both C# and Java are too low-level. Yes, they both have fully automatic memory management, and both eschew C-style pointers for the sake of safety and security, which is more than can be said for Objective-C. But the runtime that Objective-C uses to do its "object stuff" is arguably higher-level than either C# or Java, both of which still seem to cling to charmingly retro notions of compile-time optimizations and "efficiency through bondage."

Bondage. That's what non-dynamic languages feel like to me now. No duck typing? No freedom to modify and create classes at runtime? Bondage.

Sunday, October 02, 2005

APIs matter

Victor Moholy:

A bad API, like a bad novel, feels like a trick: key information is withheld and simple relationships are hard to deduce.

I had one of those "a-ha" moments reading this. ("A-ha," I thought. "I feel a rant coming on.") This is exactly why, after building probably one of the largest web applications done with OpenACS 3.2, I never managed to like the 4.x (and now 5.x) series of that toolkit.

OpenACS 4.x and later suffer from Second-system syndrome. The effect is rather Zope-like: instead of a learning curve, you have a vertical cliff to scale before you can accomplish anything interesting.

Add in bizarre and arbitrary api changes, it's no wonder nobody except the core team really got on board with the later OpenACS versions. (I suspect that the "everything must be a named parameter" had its genesis in the proliferation of functions taking over a dozen parameters. Deprecating all functions that don't require named parameters is both taking things too far, and fixing entirely the wrong problem!)

Which is a shame. OpenACS solves a much larger problem than most web toolkits attempt, giving you registration support, permissioning, etc. out-of-the-box. The 3.x series may have solved only 90% of the problem, but it did so in a much more approachable way than 4.x. (Or Zope, the only other framework that really tried to tackle this problem in that era. Don't even get me started on J2EE systems.)

Ah, well. RIP, OpenACS. It looks like Django is trying to solve some of these same problems (e.g., its permissioning support and "pluggable applications") while retaining a more lightweight approach. In this respect, Django is much more than a me-too "Rails for Python." Of the would-be Python Rails killers (and Rails itself), Django alone impresses me for attempting more than the same old, same old.

(Which reminds me -- I haven't seen this fellow's django-and-rails comparison linked anywhere, even though it's a month old. Guess I don't follow the right blogs. It's worth reading if you don't know much about one or the other, despite his lamentable lack of taste in prefering Ruby.)

Thursday, September 29, 2005

When all you have is a hammer....

Everything looks like a nail.

Some things just work better as a traditional app, guys.

Wednesday, September 28, 2005

Why friends don't let friends do J2EE

Michael Sica wrote a post about his experience writing a project manager.

Java, cool I get it. JSP, there's like 3 different ways to do everything. Which do I learn. Application frameworks, started learning Struts - what a nightmare. Stared learning JSF - what a nightmare. Found Spring and Spring MVC, and they rocked. Crap, I need to learn Tiles too. Ok, so how does Tiles work with Spring MVC. Ok that's, cool. I only need to do 6 things everytime I make a form. (I actually have a list printed out so I won't forget all the steps.)

Life's too short. Choose to be productive. Choose Python.

Tuesday, September 27, 2005

Python at Mozy.com

At my day job, I write code for a company called Berkeley Data Systems. (They found me through this blog, actually. It's been a good place to work.)

Our first product is free online backup at mozy.com. Our second beta release was yesterday; the obvious problems have been fixed, so I feel reasonably good about blogging about it.

Our back end, which is the most algorithmically complex part -- as opposed to fighting-Microsoft-APIs complex, as we have to in our desktop client -- is 90% in python with one C extension for speed. We (well, they, since I wasn't at the company at that point) initially chose Python for speed of development, and it's definitely fulfilled that expectation.

(It's also lived up to its reputation for readability, in that the Python code has had 3 different developers -- in serial -- with very quick ramp-ups in each case. Python's succinctness and and one-obvious-way-to-do-it philosophy played a big part in this.)

If you try it out, please note that after the beta period (my guess: at least a month) the "price" for mozy is that every so often we'll send you advertisements by email. (But we'll never sell your address to anyone else, and we promise not to allow any body-part-enlargement crap.) We recognize that some potential users may be turned off by this, and we will probably offer for-pay options in the future, but I don't have a time frame on when that might be.

Saturday, September 17, 2005

PyOgre

Makes me wish I had time to work on a 3D game.

An OpenLazlo blog

I'm sure being described as "an openlazlo blogger" is probably not what he had in mind, but Michael Sica of Ataraxis Software has written more about it than I've seen anywhere else. Here's his first post on the subject, from July. Around a half dozen more follow.

OpenLazlo, you will recall, is a rich web app platform that uses Jython internally. (Warning: PDF.)

(Michael's blog is also interestiing from an entrepreneurial standpoint. Starting your own company is a common fantasy for developers, and Michael is doing it. I'm catching up on the archives now.)

Tuesday, September 13, 2005

how well do you know python, part 9

(Today's questions are very CPython-specific, but that hasn't stopped me before. :)

I spent some time today looking for the source of a bug that caused my program to leak memory. A C module ultimately proved to be at fault; before figuring that out, though, I suspected that something was hanging on to data read over a socket longer than it should. I decided to check this by summing the length of all string objects:

>>> import gc
>>> sum([len(o) for o in gc.get_objects() if isinstance(o, str)])
0

No string objects? Can't be. Let's try this:

>>> a = 'asdfjkl;'
>>> len([o for o in gc.get_objects() if isinstance(o, str)])
0

So:

  1. (Easy) Why don't string objects show up for get_objects()?
  2. (Harder) How can you get a list of live string objects in the interpreter?

Friday, September 09, 2005

A review of 6 Python IDEs

(March 2006: you may also be interested the updated review I did for PyCon -- http://spyced.blogspot.com/2006/02/pycon-python-ide-review.html.)

For September's meeting, the Utah Python User Group hosted an IDE shootout. 5 presenters reviewed 6 IDEs:

(The windows version was tested for all but Eric3, which was tested on Linux. Eric3 is based on Qt, which basically means you can't run it on Windows unless you've shelled out $$$ for a commerical Qt license, since there is no GPL version of Qt for Windows. Yes, there's Qt Free, but that's not exactly production-ready software.)

Perhaps the most notable IDEs not included are SPE and DrPython. Alas, nobody had time to review these, but if you're looking for a free IDE perhaps you should include these in your search, because PyDev was the only one of the 3 free ones that we'd consider using. And if you aren't already familiar with Eclipse, PyDev probably isn't for you. (It's worth pointing out, though, that the personal editions of Komodo and Wing are only $30 and $35, respectively, and Wingware lets open-source projects use its IDE for free.)

This review first gives a comparison of features that I consider important, then gives some more subjective material for each IDE to indicate the flavor of our experience with it.

(The pronoun "we" in this post refers to the UPyUG, but "I" is always Jonathan Ellis.)

Editing

All reviewed editors provided basics like syntax highlighting. These features are not included in the comparison charts; no modern IDE should be without them.

PyDev Eric3 Boa Constructor BlackAdder Komodo Wing IDE
Keyboard Macros No Yes No No Yes Yes
Configurable Keybindings Yes No No No Yes Yes*
*Very weak UI; expect to do a lot of manual browsing
Tab Guides No Yes Yes No Yes Yes
Smart Indent* No No No No Yes Yes
*Knows to de-indent a level after break/return/etc. statements
Code completion Decent Useless No Vim-style* Good Excellent
*Can guess symbols already present in the current document
Call tips No Mostly broken No No Yes Yes*
*"Source Assistant" provides calltips and docs in a separate panel
"Go to definition" for python symbols No No No No No* Yes
*"Find symbol" is basically a find-in-files text search
Templates Yes Yes No No Yes Yes
Source Control Integration Eclipse* CVS** No No CVS/Perforce/SVN CVS
*CVS is standard; plugin availability varies for others
**SVN is ostensibly supported, but didn't work for us
GUI Builder No No Wx Qt Tk No
Emacs emulation* Poor No No No Poor Good
*None yet support VI(m) emulation -- sorry!

Debugger

PyDev Eric3 Boa Constructor BlackAdder Komodo Wing IDE
Conditional breakpoints No Yes No No Yes Yes
Evaluate arbitrary expressions Yes No No No Yes Yes
Debug external programs* No Yes No No Yes Yes
*E.g., a script processing a web server request

Miscellaneous

PyDev Eric3 Boa Constructor BlackAdder Komodo Wing IDE
Documentation Virtually none Virtually none Virtually none Poor Excellent Good
Unique features PyLint integration; "extract method" refactoring ?* Regular expression builder ?* Multilanguage; save macros; regular expression builder Source Assistant; scriptable with python
*I didn't notice anything worth mentioning

Impressions

PyDev

If you like Eclipse, PyDev is a decent choice (provided, of course, that you have a suitably beefy machine). If you are unfamiliar with Eclipse, good luck with the learning curve; you'll need it.

PyDev is the only IDE reviewed where features such as code completion will not work unless you add the .py files to the PyDev project.

"Extract method" is cool, but limited. Perhaps the biggest drawback is, it doesn't know its limits and will happily perform invalid refactorings.

Code completion is the best of the free IDEs we reviewed.

Eric3

Eric3 has a lot of good features. These are overshadowed by two problems: a horribly cluttered UI, with a correspondingly painful initial experience, and the worst implementation of code completion I have ever seen.

You can choose from two kinds of code completion: completion for the stdlib, and completion for your current project. You cannot have both at once. "What the hell?" I hear you say. Yes, but it gets worse: say you decide to pick the stdlib. You write, "import os; os." and wait expectantly for the completion. You get... a list of all symbols anywhere in the stdlib! Or pretty close to it. Wow.

Calltips are a similar mess. If you write "os.path.exists(", you may get the calltip for os.path.exists, or you may get the calltip for any other exists method.

Eric3 is documented as well as the other open-source projects we looked at, which is to say, there are some screenshots and a mailing list.

Boa Constructor

Supposedly, Boa Constructor supports code completion and call tips, which you can invoke with control-space.

This didn't work for us. At all. We didn't spend too much time trying to figure out what was wrong, though, because the worst part of the experience was how amazingly unstable Boa Constructor is. (Well, "amazingly" except to anyone who has used earlier versions of BC, I guess. Apparently this isn't new with the 0.4.4 release.) It would corrupt dialogs, screw up components such that they had to be deleted and re-added, and crash to the desktop. I hadn't considered rating IDEs on MTBF before, but Boa Constructor made me think hard about it.

Oh, and if you try BC and your first component on the frame sizes to take up the entire frame once you run it -- that's normal, apparently. Just add more components and they will start behaving. Until it crashes.

BlackAdder

The only IDE here that doesn't support code folding, BlackAdder fails in more important ways as well. (Hey, I almost never bother with folding even in other IDEs.) Most glaringly, it was the only reviewed IDE that we could not get to run a test program; it complained instead about DLL import problems. Possibly this is due to us using PyQT 3.1.14 instead of 3.7, but the former is the only one available to download from TheKompany's site. (Trying to install on Linux resulted in even less success; we never did manage to get it to run.)

BlackAdder also has the uniquely annoying trialware behavior of closing itself after 10 minutes no matter what you are doing. (The description on thekompany.com, which says that saving is disabled, is incorrect -- you do get to save your work.)

Vim-style code completion is pretty weak stuff when you're competing with companies and free projects that do the real thing.

Editor doesn't obey normal conventions like double-clicking on an option to pick it and close the dialog or re-opening a file when it changes on disk.

The only documentation is a 35-page PDF, about 18 pages of which is actually reference material. The rest is taken up by installation instructions (see above for how helpful this was to us), a tutorial (fairly useful) and a guide to QT Designer. It's hard to complain about the reference material since there really aren't a whole lot of features to document, but it's still underwhelming. (No mention is made of code completion, for instance; I never would have known about it if I hadn't seen a reference in some Usenet thread.)

BlackAdder is the oldest IDE reviewed here (1.1 was released in 2003), which speaks in more than one way to its relative priority for TheKompany. I'd be embarassed to leave a product with this level of problems in my store.

Komodo

Komodo is a good IDE, but it has rough edges. Its code completion engine is prone to refusing to help out in certain files. I couldn't discern any pattern in the set of files it couldn't figure out. It also occasionally reports parse errors on valid code; this, for instance, is valid Python 2.4 (which Komodo 3.1 supports), but even in a blank file results in a parse error:

  other_space_used = sum(max(v.space_used, v.pending_space_used)
                         for (k, v) in u._machines.items()
                         if k != self.machineid)

Another one: Alt-b cannot be rebound to anything. I filed a bug report against Komodo 3.0 for this over a year ago, and it's still there in 3.1. (I also contributed to a report listing ways to make Komodo's emacs mode not suck quite so much out-of-the-box. This hasn't improved since 3.0.1, either. I guess it's fair to say I'm not very impressed with ActiveState's support system.)

Probably the biggest win for Komodo is its support for multiple languages. If you're not lucky enough to only code in Python (I am), this might be the killer feature that makes you an ActiveState customer.

Wing

Wing shares with PyDev the distinction of being the most heavyweight IDE in this review, in the sense of needing a fairly beefy CPU to remain snappy. (Two data points: on my 1.6 GHz Pentium M with 512 MB of ram, it's fine; on my 900 MHz P4 -- it's a long story -- with 512 MB, it's noticably sluggish.) This may be due to the bulk of Wing itself being written in Python.

I called out BlackAdder for having an obnoxious trial behavior, so I should probably point out that Wing has the least obnoxious trial. You don't have to enter your email address (BlackAdder/Komodo) or download and run an executable license key (Komodo); you just install the software and Wing asks if you would like it to contact wingware.com to auto-install a trial key. Slick.

Wing is the only IDE here to give you code completion for more than function and class variables; it also completes keywords, modules (in import statements), and locals. This is more useful than it sounds; once you get used to it, you won't want to go back. (For my .NET readers: Wing does Whidbey-style completion.)

Wing's Source Assistant is nice, but it's distracting to have to look at a different panel to get call tip information. Better to have both.

Wing is the most polished of the IDEs here. I suspect this is due to the Wingware developers "dogfooding," using Wing to develop Wing. There are fewer oversights and more of the small touches that indicate quality. A minor exception is that the documentation has become outdated in some places.

Let me also note that if you tried and rejected Wing 1.1 back in the day, you should give Wing 2.0 another look. I fall into that camp myself; on Windows, at least, Wing 1.1 was horribly ugly. I couldn't stand looking at it. Wing 2.0 is much, much easier on the eyes. In fact, almost all the IDEs we reviewed look pretty good. Eric3's cluttered look is the only exception.

When I've had occasion to send several questions ("How can I make enter autocomplete, as well as tab?") and suggestions to the Wing IDE mailing list, Wingware has responded promptly. ("Use control to select multiple keys in the autocomplete dialog." Okay, I guess that was a stupid question.)

Conclusion

Early last year, I bought Komodo Personal edition for $30. That's a pretty good deal, especially when you consider its support for other languages (Perl, PHP, and TCL) as well as Python. I still think Komodo Personal is a good deal, but today I would go with Wingware Personal ($35) instead, primarily on the strength of its better code completion support, "Go to definition" feature, and Source Assistant. Superior Emacs emulation (superior to just about any other non-Emacs editor I've ever used, actually; I suspect Wingware has at least one Emacs refugee) seals the deal. A few days ago, I ordered Wing IDE Professional (paid for by my boss).

If you need an integrated GUI builder, or you have an older machine, Komodo remains a good choice, although not many people these days would pick Tk as their first choice for a GUI toolkit.

Of the free choices, PyDev is the clear choice if you have Eclipse experience. If not, well, the situation isn't pretty. Perhaps you'll have better luck with one of the IDEs we didn't review here.

Addendum

ActiveState, TheKompany, and Wingware were all kind enough to donate a license for their respective IDEs to the Utah Python User Group as door prizes for the meeting where these reviews were conducted. (None of the reviewers received any of these.) We thank these companies for their support. (Sept 22: Shawn Gordon of TheKompany appears to have decided not to honor his commitment here.)

Thanks also to all our reviewers. Alphabetically, these are Brent Hughes, Byron Clark, Jason Reusch, Jonathan Ellis, and Justin Wilson.

Update Sept 10: noted that Eric3's svn support didn't work for us

Wednesday, August 24, 2005

Code Jam 2005

I tried my hand against around 5000 other programmers in Google's Code Jam 2005. Last year Windows corrupted the hell out of itself halfway through, when I had foolishly not saved any work. The time and concentration lost rebooting killed me. (I had the infamous anti-aliasing problem for my 500pt question. The lowest qualifying score from that group was in the 170s, IIRC. Google dialed down the difficulty considerably this year.)

This year I went into battle armed with Visual Studio 2005. TopCoder doesn't recognize any of the beta additions, but Studio 2005 has a pretty decent Emacs mode. Instant 10% productivity bonus!

The alert reader will have already noted the use of the past tense in the first sentence and concluded that I didn't make it into the top 500 to qualify for the next round. Correct; I missed the cut by 4 points (732 to the 100th place guy's 736).

Ouch. I'd kinda rather have been way outclassed than lose by 4 points out of 1000. Now I'm kicking myself for wasting time checking and re-checking that I hadn't missed something in the 750 pt problem before submitting. It was just so much easier than last year's 500 pt question that I thought I couldn't have solved it that quickly. Having to shake the rust off my C# (what do you mean, you have to declare types? That is so last century!) didn't help either.

Maybe next year they will have Python and the top spots won't be so completely dominated by C++ people with dozens of topcoder-specific macros. Grr.

Wednesday, August 17, 2005

Durus 3.0 alpha released

Thought this was worth pointing out since the Durus guys don't seem to make PR much of a priority. (No slur intended. You could say the same thing about Spyce. :)

Durus is a python object database similar to ZODB, but (much) simpler. Their Pycon 2005 paper is a good intro.

Here's the 3.0 changelog. IMO the coolest parts are zlib compression and incremental packing.

Spyce has a Durus interface (spyDurus) that makes persistence even simpler than the various ORM tools. The to-do demo uses Durus.

Sunday, August 14, 2005

A little PHP humor

(Seen in a slashdot comment.)

Possible mottos for the PHP project:

PHP: We'll be there for you if your development environment doesn't have enough side effects.

PHP: Because we know the money's in the maintenance contracts.

PHP: Because you obviously don't know any better.

PHP: We take security as seriously as Microsoft ten years ago.

PHP: Doing it fast is always better than doing it right.

PHP: Proving that if any idiot can write an e-commerce package, any idiot will.

PHP: Yet another great reason to make regular backups.

PHP: Fast, cheap, and robust. Two out of three ain't bad, right?

Sunday, August 07, 2005

Guido's OSCON Python keynote

Guido has posted his powerpoint slides in the usual location. (Opens fine in OpenOffice.) The slides are mostly the same as the ones from EuroPython a month ago; here are the main differences:

  • Some odd slides during the introduction (anyone who was there care to explain these?)
  • language data scraped from sf.net -- or you could just look at this page; no need to scrape and piss off SF for something this simple. (Poor Dylan -- even less projects than Cobol -- 2 vs 4.)
  • "What does this buy us" slides added to 342 (generator) and 343 (with) PEP discussion (highly recommended if you haven't been following python-dev religiously)
  • Exception reform discussion

Personally, the "with" statement excites me the most. The exception cleanup is overdue but less sexy. :)

The static typing subject has been talked to death but I'll add my two cents.

Basically, it's a bad idea because it kills duck typing by requiring interfaces like "iterable"--a horrible idea even if all stdlib classes grow corresponding interfaces. The repeated realizations that I can't use a library because its author didn't bother making interfaces for the basic classes involved is a large reason I don't feel comfortable in static languages anymore.

"Adaptation" solves that problem in theory; in practice, having to create adapters to convince Python to treat my objects as ducks would be a noxious imposition. Code that exists only to convince the runtime that I know what I'm doing is un-Pythonic.

On the other hand, I think it would be misleading for "x: file" to not perform any assertions at all, and therefore also a bad idea. Better than inflicting interfacitis on Python, though.

Friday, August 05, 2005

Open-source, subversion, mono, and gcc

I read a blog entry about a subject dear to my heart: migrating from cvs to svn. I know svn isn't perfect, but it's just so much better than cvs that it's no contest. If you have objections to svn that are even just a few months old, they have probably already been fixed. One of the first things I did at my new job was to perform the migration and everyone (well... all one other developer who was hired before I was and is still there) thinks it was a great move.

But I digress. Sort of.

Reading said blog, I thought, "Ha, cool! Those Mono guys whined about svn blame being slow, but then those manly GCC guys dug in and fixed the problem instead of whining some more!" What a poster-perfect picture for the virtues of open-source, huh?

Except, while looking for more details, it turns out that the developer responsible for the fix, Daniel Berlin, was a svn developer from way back. As well as (possibly more recently) gcc. (Does this guy ever sleep?)

So it's a little less heroic than it initially appeared, but still, not something you'd see in the closed-source world. "Hey, Microsoft: that bug in Excel, the one you weren't going to get to for a while? My buddy used to work on the Excel team. Mind letting him fix it for me?" Right.

If anyone knows a more complete/accurate summary of what happened here, my curiosity is piqued. The best I could find is from early in the process and not very detailed.

NagleQueue

Here's a piece of code for the one or two other developers writing intensely network-dependent code in Python. The idea is, instead of a protocol that looks like

[command][arg]
[command][arg]
...

You save overhead (in your protocol and in the network's, if you're doing one connection per command) by batching things:

[command][arg][arg]...

Pretty obvious stuff, no doubt. But I thought the following class makes it rather elegant.

class NagleQueue:
    """
    When an item is put, NagleQueue waits for other puts for
    aggregate_time seconds (may be fractional) and then calls
    aggregate() with a list whose maximum length is max_items.

    TODO: currently, NagleQueue always waits the entire aggregate_time,
    even if max_items items are added sooner.

    NagleQueue starts a thread to handle the aggregate() call;
    this is not a Daemon thread, so you must call stop() before your
    application exits.  NagleQueue will process any remaining items,
    then quit.

    It is an unchecked error to put additional items after calling stop().
    (They may or may not get processed.)
    """
    def __init__(self, aggregate_time, max_items):
        self.q = queue.Queue()
        self.running = True
        self.aggregate_time = aggregate_time
        self.max_items = max_items
        threading.Thread(target=self._process, name='naglequeue').start()

    def put(self, item):
        self.q.put(item)

    def stop(self):
        self.running = False

    def _process(self):
        while True:
            try:
                item = self.q.get(timeout=1)
            except queue.Empty:
                if not self.running:
                    break
                else:
                    continue
            time.sleep(self.aggregate_time)

            L = []
            while True:
                L.append(item)
                if len(L) >= self.max_items:
                    break
                try:
                    item = self.q.get_nowait()
                except queue.Empty:
                    break
            self.aggregate(L)

    def aggregate(self, items):
        """
        combines list of items into a single request
        """
        raise 'must implement aggregate method'

Wednesday, August 03, 2005

Python at the .NET user group

I was invited to present on Python and IronPython at the Northern Utah .NET User Group last night. I spent the majority of the time on Python itself, guessing (correctly, with the exception of a couple Zope users) that most people wouldn't know anything about Python beyond having heard of it.

There was a lot of participation. It ended up lasting about 90 minutes, with a lot of Q&A throughout. I enjoyed myself and I think most of the .NET guys did too.

Thanks for the chance to speak! A PDF of my slides is up here.

Thursday, July 28, 2005

MochiKit 0.5

MochiKit is a platform-agnostic javascript framework that doesn't suck. (Here's the release announcement, following the announcement of a public subversion repository a few days ago.) The documentation alone should make MochiKit the preferred js library. It's a huge improvement over the alternatives.

Bob Ippolito has been working on MochiKit for a while. I've learned a lot from his javascript blog posts as he's worked on it. If you think you know javascript, but don't know what a prototype is (I didn't), you need to read his blog.

(Yes, Bob knows about JSoLait and Prototype. He doesn't like them, and gives excellent reasons for the genesis of MochiKit.)

Wednesday, July 27, 2005

iXP

I love Mike Spille's blog. He doesn't post terribly often, but when he does it's always worth reading. Here he is with a savagely funny mock of XP.

Rule 0. Whenever you type an 'i' - drink!. Also known as the zeroeth iXP rule. While the vision began with Alternate Letter Pair Programming Order, mentioned below, it's really this rule that made iXP gel, and which differentiated it from previous similar practices.

Thursday, July 21, 2005

Spyce testimonials

Since the Daily Python URL was kind enough to link to the 2.0.3 announcement, I thought a few testimonials might be in order. :)
I tried downloading, installing and using Spyce 2.0.2 on Windows and Linux yesterday and it worked like a dream. I have already set up a useful little dynamic site on a linux box and plan to expand it radically in the weeks ahead. I am using Spyce in webserver mode and also using the scheduler to trigger periodic repository updates.
    --Rock Howard
Spyce is really speeding up my "project"... at this rate, I'll be ready for pre-beta testing in about two months. I'm utterly addicted to encapsulation via active tags.
    --Tim Lesher
(Not to pick on them since they do have a nice framework, but Tim was formerly using CherryPy. Most recently, anyway.)

Spyce 2.0.3 released

2.0.3 is a bugfix and documentation-improvement release. The installation section of the manual has received particular attention. There is also the new section on starting your first project, which answers the FAQ, "how do I organize my Spyce files?"

Changelog:

    - fix pool bug if server not in concurrency=threaded mode; reported by "Dude"
    - documentation improvements
    - avoid stomping on user python modules named 'config'; reported by Betty Li
    - (John Reese) fixed bad interaction of "redirect if directory path doesn't 
      end in /" and "look for default index.$[index extensions] files" code
      in Spyce webserver
    - default concurrency mode for Spyce webserver is 'threading' instead of None
    - spyceProject script to automate new-project creation

Update: you may also be interested in the new testimonials post.

Wednesday, July 20, 2005

on Broken Pipe and Connection Reset By Peer

Maybe I'm just dumb, but I always thought "broken pipe" meant, "the other end of this socket closed before I finished sending something" and "connection reset by peer" meant, well, roughly the same thing. (As well as indicating some slightly more esoteric problems.)
Turns out though, "broken pipe" actually means "I just tried to send something and the socket was already closed to sending."
So in the following example, if the other end of (TCP) socket "sock" closes or dies before the write method, "connection reset by peer" will be raised. The next write will give a broken-pipe error, since the socket now knows that further sending is invalid.
try:
    sock.write('foo')
except:
    pass # connection reset by peer
sock.write('bar') # broken pipe


Sunday, July 17, 2005

Jython is back

After almost two years with no releases, Brian Zimmer has released an alpha version of Jython 2.2 (incorporating some of the patches I submitted back in Jan 04). Ha!

Brian, you will recall, got a PSF grant for Jython and started work earlier this year. Obviously his original estimate called for a much faster release, but that's how software development goes. Six months to a release after inheriting a new-style class branch that I suspect was mostly broken really isn't too shabby, especially considering how long it took the CPython team.

I haven't been following Jython development since I gave up hope of Samuele Pedroni ever getting a release out last year, but it's clear that Jython isn't a one man show anymore. Frank Wierzbicki and Clark Updike have been comitting code; almost certaintly there are others too that I didn't immediately notice. Frank and Clark have both been involved with Jython for a while, but new developers are getting involved too. Good to see.

Big thumbs up to all involved.

Wednesday, July 13, 2005

How well do you know python, part 8

Here is a mostly-functioning class that facilitates writing producer / consumer algorithms. It is designed to support a single producing and multiple consuming threads, i.e., scenarios where consuming involves some blocking operations such as communicating over a socket.

"Mostly," in this case, means that there are two bugs in this class that can be fixed in a total of four or five lines. One is arguably more subtle than the other, but they involve the same part of the code. Can you spot them?

Warning: this one is tougher than part 2, which also dealt with threading.

import Queue as queue
import threading

class PCQueue:
    def __init__(self, initialthreads):
        self.q = queue.Queue()
        self.running = True
        self.condition = threading.Condition()
        for i in range(initialthreads):
            self.add_consumer()
    def run_consumer(self):
        while True:
            self.condition.acquire()
            try:
                self.condition.wait()
                if not self.running:
                    return
            finally:
                self.condition.release()
            obj = self.q.get()
            self.consume(obj)
    def put(self, obj):
        self.condition.acquire()
        self.q.put(obj)
        self.condition.notify()
        self.condition.release()
    def consume(self, obj):
        raise 'must implement consume method'
    def add_consumer(self):
        threading.Thread(target=self.run_consumer).start()
    def stop(self):
        self.running = False
        self.condition.acquire()
        self.condition.notifyAll()
        self.condition.release()

Here's a short example using PCQueue.

class PCExample(PCQueue):
    def consume(self, url):
        import urllib
        n = len(urllib.urlopen(url).read())
        print '\t%s contains %d bytes' % (url, n)

q = PCExample(2) # 2 threads
while True:
    print 'url? (empty string to quit)'
    url = raw_input()
    if not url:
        break
    q.put(url)
q.stop()

Friday, July 01, 2005

Two Wax projects for Summer of Code

The PSF put up a list of projects accepted for Google's Summer of Code. Not one, but two projects deal with improving Hans Nowak's Wax toolkit. Very cool: hopefully by the time I have to write another desktop-bound application, Wax will be mature and well-documented. Tkinter is too underpowered, and wx is too clunky.

That's why I added Wax to the SoC coding project ideas wiki, although it wasn't immediately clear if this was welcome. Glad it worked out.

(Two people working on Wax is old news if you read Hans's blog. Which I haven't been, inexplicably. Ah, well.)

Thursday, June 30, 2005

Of all the things I've lost...

I installed a sys.excepthook in my project at work that uses the logging module to record any uncaught exceptions. It didn't work.

I did what any lazy programmer would do: ask someone more experienced. "Do you have to set sys.excepthook in each thread?" I asked, more-or-less. "I don't think so," he replied.

Hmm. Maybe the developer I'd taken over from was setting excepthook later on in the initialization or somewhere else. ... Nope.

Finally I wrote this:

import sys, threading

def log_exception(*args):
    print 'got exception %s' % (args,)
sys.excepthook = log_exception

def foo():
    a = 1 / 0
threading.Thread(target=foo).start()

Playing with this a bit demonstrates that sys.excepthook doesn't work in subthreads, at all. The documentation doesn't mention anything of the sort. Smells like a bug to me. (I did file one.)

I belatedly googled "sys.excepthook threads." (I must have been a bit slow this morning to not do this first.) I was more than a little surprised to see my name in the first result that came back. (And a little disappointed that eight months later there's still no replies. :)

I've been a father for a couple years now, so I've gotten used to my memory being a little fuzzy now and then. But usually it's just, well, a bit fuzzy. I can tell that there are details I don't remember immediately, and they usually swim back if I think about it hard enough. But I don't remember posting that at all, nor can I recall what I was working on to prompt it. Funny. But also a little scary.

The tale of a wiki diff implementation

I run a web game that I started in late 2000, back in the Dark Ages before there was a decent python web toolkit. It runs on a then-current version of the OpenACS TCL-based toolkit. (at around 20kloc, porting it to a more modern system wouldn't be worth the effort now.)

Recently, a player suggested that I add a wiki, since my own documentation is chronically out of date and player-run sites tend to suffer bitrot as well. (How many games are you still playing that you started 5 years ago?) So, I backported a modern OpenACS wiki module -- no trivial task; a LOT has changed in OpenACS, and not all for the better -- and was set. Except the module I backported didn't have diff functionality, probably because the various TCL options mostly suck.

Enter TclPython, a tcl module by Jean-Luc Fontaine (who is obviously a far better C hacker than I) that embeds a python interpreter. Sweet! My life just got a lot easier:

package require tclpython
set py [python::interp new]

# $a and $b are the text of the two revisions -- inject them into the python interpreter
$py exec "
from difflib import HtmlDiff
a = \"\"\"$a\"\"\".split('\\n')
b = \"\"\"$b\"\"\".split('\\n')
hd = HtmlDiff(wrapcolumn=80)
"
ns_return 200 text/html [$py eval "hd.make_file(b, a, context=True)"]

The finished result used make_table and included some css to make things pretty, but that's all there is to it fundamentally. Thanks, Mr. Fontaine!

Wednesday, June 15, 2005

SF.net examples fixed, again

This time I apparently broke it by copying the development spyceconf.py, which among other things references the authentication tag that doesn't exist in 2.0, into the sourceforge site. Oops. (The 2.0.2 release itself wasn't affected.)

Monday, June 13, 2005

I figured out why Python's threading library bugs me

Reading Aahz's 2001 OSCon presentation, I ran into a slide that crystalized it [paraphrased]:

  • Perl: There's more than one way to do it
  • Python: There should be one (preferably only one) obvious way to do it
  • Python's threading library is philosophically perl-ish

That pretty much says it all. Well, that and the main classes are (still) virtually undocumented.

Update: I'm referring to the synchronization classes in this module, not the Thread class, which is straightforward enough.

Saturday, June 11, 2005

Why PHP sucks

(July 8 2005)

Apparently I got linked by some PHP sites, and while there were a few well-reasoned comments here I mostly just got people who only knew PHP reacting like I told them their firstborn was ugly. These people tended to give variants on one or more themes:

  • All environments have warts, so PHP is no worse than anything else in this respect
  • I can work around PHP's problems, ergo they are not really problems
  • You aren't experienced enough in PHP to judge it yet

As to the first, it is true that PHP is not alone in having warts. However, the lack of qualitative difference does not mean that the quantitative difference is insignificant.

Similarly, problems can be worked around, but languages/environments designed by people with more foresight and, to put it bluntly, clue, simply don't make the kind of really boneheaded architecture mistakes that you can't help but run into on a daily baisis in PHP.

Finally, as I noted in my original introduction, with PHP, familiarity breeds contempt. You don't need years of experience with PHP before an urge to get the hell out and into a more productive environment becomes almost overwhelming -- provided that you have enough experience to realize what that nagging lack of productivity is telling you when you go to consult the documentation for the fifth time in one morning.

Basically these all boil down to, "I don't have enough experience to recognize PHP's flaws because I haven't used anything better." Many years ago, I had this same attitude about Turbo Pascal: it was the only language I knew at the time, so anyone who pointed out its flaws was in for a heated argument.

There's nothing wrong with being inexperienced, as long as you have an open mind. If you do, try Spyce. Try RoR, if you must. Better toolkits are out there, for those who aren't satisfied with mediocrity.

I've done a fair bit of web development. I've written HTML-generating code in C CGI scripts, Cold Fusion, TCL (with OpenACS), ASP.NET, and, of course, Python with Spyce.

Two technologies I've steered clear of are any J2EE stack and PHP. I've seen enough of each to know that I'd be immensely frustrated with either. Briefly, although they are quite different, neither is elegant, and elegance counts.

Recently, though, I had to spend a few days extending a small amount of PHP code. (I'm just glad for two of those qualifiers: "few" and "small.") The more I used it, the less impressed I was, which is why I don't think a longer experience would make this post any more favorable to PHP.

This is far from an exhaustive list. It may have some reasoning in common with other voices of reason, but I'm writing this primarily from a Python developer's perspective. So, I won't waste time bashing PHP for being dynamic, which some Java zealots do; nor will I fault it for mixing code and markup, which also has its uses (and Spyce recognizes this). PHP's problems are much, much deeper.


First, let's try to be a bit more specific. Does PHP the language suck, or does PHP the environment suck?

They both suck.

In fact, they suck for the same reason: PHP-the-language and PHP-the-environment both grew by accretion of random features, not by any purposeful design for orthogonality. So you have idiot "features" like magic quotes ("Assuming it to be on, or off, affects portability. Use get_magic_quotes_gpc() to check for this, and code accordingly) and register globals (same disclaimer applies, only more so).

Trying to write "portable" php code is such a disaster that it's no wonder almost nobody tries; you can't even code for a least-common-denominator version because (a) so much is subject to change on the whim of the site's config file and (b) even if it weren't, the PHP designers change the defaults almost as often, even within minor version releases. (E.g., the registerglobals change for 4.2.)

The PHP community realizes this to a degree, even if the fanboys won't admit it. PHP5 uptake is almost as glacial as MySQL4 was/is because it breaks so much code. One of the biggest ways is, "all your copy-on-assign code, isn't anymore."

That bears explaining if you haven't coded in PHP4: any time you make an assignment in PHP4, what other languages would call a "deep copy" is performed. So you might naiively expect this code to add a new key/value pair to the associative array at $a[0]:

<?
$a = array(array('foo' => 'bar'));
foreach ($a as $item) {
   $item['new key'] = 'new value';
}
print_r($a);
?>

This, of course, changes $a not at all, because the assignment to $item is a deep copy. (It's also slow as hell if the items you're deep-copying are substantial.) The workarounds for this are truly ugly.

This changes completely in PHP5, which is a good thing, unless you're trying to upgrade an existing body of code. That's not a small "unless;" if you weren't using PHP4, there's really no excuse to start out in PHP5 instead of Spyce or CherryPy or Rails.

Even with a willingness to break backwards compatibility, a lot of broken behavior persists in PHP5. For instance, since I brought up PHP arrays: an array in PHP is really a map. When you're using it as a linear array, it's really mapping keys 0, 1, etc. to your values. (I don't want to know what it does when you're using it as a 2D array.) This might seem like a good idea, until you spend about two seconds thinking about it, at which point those of you who have had a basic introduction to data structures will be thinking, "What the hell? The performance will SUCK!" And you are entirely correct. The only other language I can think of that does this is AWK, and it was a bad idea there, too, but less of a problem since, well, when was the last time you wrote an AWK script longer than 10 lines?

PHP-the-language also shares with Perl the unfortunate tendency to guess what the programmer "really meant" instead of raising errors. (String where an integer makes more sense? No problem, we'll just throw in zero!) Unlike perl, there's no "use strict" option to mitigate this.

I could keep going, on how, for instance, the PHP environment doesn't give you any way to have a single, multithreaded long-running process, which is probably why you see so much PHP code that sticks everything into the session object. Or how PHP's string processing library adopted the C stdlib functions without improving on them. Or a dozen things, but a comprehensive enumeration of PHP design flaws would be a daunting task indeed.

In short, PHP sucks because, PHP's authors are prone to confuse "pragmatism" (a fine design goal, if done well) with "adding random features without considering how they impact the language as a whole." Thus, its authors have found it necessary to correct obvious flaws in both minor and major releases, with the result that the recent PHP5 breaks with the past to an unprecedented degree while still leaving many fundamental flaws un-addressed. I don't know if this is because they didn't recognize those flaws, or more likely, because they were willing to impose "requires a lot of pain to upgrade" but not "requires a complete re-write."

Ian Bicking wrote that "Python could have been PHP" (in popularity for web development). If I were a PHP developer, I'd be pretty disenchanted with how the language has evolved; I don't think it's impossible that Python could have a second chance.

Tuesday, June 07, 2005

Anders Heljsberg doesn't grok Python

While Debian was releasing Sarge and Steve Jobs was introducing MacX86 yesterday, Anders Hejlsberg spoke on C# at Microsoft Tech-Ed. James Avery writes:

After a couple questions from other people I was able to get in the other question I was dying to ask. What does Anders think about the resurgence in dynamic typing from languages like Python. Basically he said that he understands what benefits people are getting from dynamic typing, but he thinks they can get the benefits of dynamic typing without sacrificing strong typing. He talked about inferring type (what anonymous methods do now with delegates) and how that might be a way to get the coding speed and ease without sacrificing the strongly typed information.

Unfortunately, omitting type declarations is only a small part of Python-esque dynamism. One of the smallest, in fact. Far more important are the ability to modify objects and classes at runtime, which allow you to do things in Python that would require code generation (which is fragile, at best) or AOP language modifications to do in C# or Java.

Equally important is the attitude this fosters in the Python community: "Python assumes we're all consenting adults." "Private" restrictions are discouraged (and can still be bypassed by others if they're willing to do less work than performing ordinary refleciton requires in other languages.) As Sion Arrowsmith recently put it,:

Years of writing and maintaining others' C++ and Java code (plus one year of maintaining Python code and rather more writing) has led me to believe that there is no justification for truly private variables... [D]enying derived classes full access to your inner workings just leads to clumsier (less readable and more bug-prone) implementations derivations... [I]t's based on the hubris that you are a better programmer than anyone who might want to extend your class and can forsee all circumstances in which it might be subclassed.

This attitude of "we know better than you what code you will want to write" is found in the Java world, but it's really pervasive at Microsoft. C# methods are "final" by default; no polymorphism for you unless the original author was generous enough to specify "virtual!" This can be done in Java, but at least it's not the default. Also, huge parts of the runtime (in both .NET and Java, but more so in .NET) are "sealed," preventing subclassing entirely.

You couldn't inflict that on your users in a language like Python. Even if you could "seal" a Python class, which will never happen since it's a horrible misfeature, a user could still create his own class with the right attributes and nobody would know the difference (because of duck typing, another important part of the we're-adults-here philosophy). Well, unless you littered your code with isinstance calls, but nobody would use code THAT poorly written... if you want to play in Microsoft's world, you don't have a choice.

Wednesday, June 01, 2005

Spyce 2.0.2 released

Second bugfix release. Get it here.

Changelog:

    session_dir uses config.tmp by default if no directory is specified
    fix for session_dir pickling on win32
    fix for fileCache pickling bug on win32 reported by Jaros³aw Zabiello
    fix for sessions + handlers problem reported by Jonathan Taylor
      * all module init() methods are now run before handlers are called.

Tuesday, May 31, 2005

How well do you know Python, part 7

Given the following two classes,

class A:
    def foo(self):
        print self.__class__.__name__

class B:
    pass

why does this work,

def bar(self):
    print self.__class__.__name__

B.foo = bar
b = B()
b.foo()

while this does not?

B.foo = A.foo
b = B()
b.foo()

Tuesday, May 24, 2005

Utah Python User Group

The former Provo-Orem Python Meetup Group is now the Utah Python User Group. I've set up a rudimentary website and discussion list; anyone interested in Python is welcome to join the list and/or attend the monthly meetings (next one on June 9th). (We're already on the user group wiki in case utahpython.org is hard to remember. :)

Friday, May 20, 2005

Spyce 2.0.1 released

Spyce 2.0.1 is available over at the usual place.

This is a bugfix release. Changelog:

    use tempfile.gettempdir() as default config.tmp
    add originalsyspath to spyceconf
    bugfixes for Active Tag compiler
    portability & other bugfixes (John Reese)

Demos working again on sf.net

SF changed the web space over to read-only. I updated the Spyce install to use the /tmp/persistent space and things are working again.

Thursday, May 19, 2005

Spyce vs ASP.NET

A former co-worker asked me over IM,

What advantage, other than being able to use it on a Linux box, does [Spyce] have over the ASP.NET Model? Because I can tell you, with Visual Studio, I feel pretty productive.

I answered,

The main one is a significantly shallower learning curve. Once you know how the ASP.NET event model and page lifecyle work, you are set -- but you remember how "easy" that was for [our] students. With Spyce there is only one rule: handlers run before the rest of the page, in the order they were added. That's it.

And, of course, Spyce runs Python. This is significant: Python is addictive; once you start writing code in it, it's painful to use less-expressive languages. This is something most people won't get until they try it, unless maybe they are old-school Lisp hackers.

Now, Spyce isn't quite to the point where it matches ASP.NET feature for feature. But it's close, and getting closer. I checked in an authentication Active Tag earlier today that will be in Spyce 2.1. After that, the only thing I miss from ASP.NET is the (form) validation controls. That will probably also be in 2.1, and then what is Spyce missing? ADO.NET? No, thanks; any feature that has entire books devoted to it has a very suspect cost:benefit ratio. I'd rather use an ORM tool like PyDO. Master pages from ASP.NET 2? Already done. Weird-ass bugs caused by IIS misconfiguration? I think we can leave those out.

Microsoft ahead of Sun in dynamic language support

Microsoft's Chris Anderson writes:

...our singular focus on strongly typed compiled languages has blinded us to the amazing productivity and approachability of dynamic scripting langauges like Python and Ruby. I'm super excited that we are changing this. Hiring Jim Hugunin [to work on IronPython is a great start.

Jython has long been the only Python (or any major dynamic language) implementation on a Big Platform VM, and it's still substantially more usable than IronPython despite months/years of languishing while Samuele Pedroni as the sole "active" project admin wrote little code and checked in less, much to the dismay of people who wanted to work on Jython. (Not that I'm bitter.) Now that Brian Zimmer got a PSF grant to work on Jython, thinks are finally moving again, slowly.

But despite the moral support from Sun managers like Tim Bray, Sun really hasn't done much for dynamic languages on the JVM. Sure, they hosted a meeting last year on the subject, and they support scripting their NetBeans IDE with Jython and Groovy, but that's extremely lukewarm support compared to Microsoft's hiring two developers full-time to work on IronPython.

The party line, I've heard, is that Sun doesn't want to show favortism to Jython over, say, Groovy. Which is a crock; Jython is stable and fairly widely used, as such things go; Groovy may not be quite the unholy mess it was earlier this year but it's clearly got a few years to go before it gets to a Jython-like level of maturity. If Sun were interested, the obvious thing is to support Jython now and Groovy, JRuby, etc., when there is sufficient community interest.

Which is why if my current employer drops Python in favor of JSP, as seems likely, I'll probably take a .NET gig over a Java one if I can't find more directly python-related employment.

Monday, May 16, 2005

Spyce 2.0 final

Spyce 2.0 is available here.

"You've taken spyce off my 'disgusting template engines' list. This is how I'd want to see apps structured, exactly."
    -- Tim Lesher
"I feel easily five times as productive in Spyce as I do in JSP."
    -- Conan Albrecht

Spyce is a python web application server, combining the features of popular frameworks such as JSP and ASP.NET with Pythonic elegance. Spyce may be deployed as a standalone server (or proxied behind Apache), or under mod_python, FastCGI, or CGI. Documentation and demos are here.

Highlights of Spyce 2.0 include:

Thursday, May 12, 2005

Trial periods for software developers

Tim Lesher has an interesting post about applying meritocratic principles to hiring software developers. He suggestion is to hire developers as contractors for a trial period before taking them on full-time.

If you as a hiring manager aren't sure about someone, either on a technical level or just fit with your team, this makes sense. I didn't have precisely this experience when I started as an instructor at Northface, but I did start at a lower salary with an explicit "if you're still here in 3 months, here's what your salary will rise to." (This was primarily because they wanted someone to teach ASP.NET, with which I had no experience at the time.) I thought this was a pretty fair approach, and much better than a rigid "if you don't have the right keywords on your resume, don't bother no matter how good you are."

But companies won't bother giving the "maybe" candidates a trial period if they think they can find a "yes" candidate in a reasonable time. Not out of malice, and not even out of stupidity: even with a good candidate, they know that net productivity will decrease temporarily while one or more of the existing team answer the new person's questions and otherwise help him get up to speed; the bigger the project, the longer this lasts. With a "maybe" candidate, the odds that the eventual increase will make up for that only get worse.

Companies with smaller projects can better afford the risk, but they will still prefer to hire someone they are sure will be an asset than take chances.

Later at Northface I did the technical interviews for instructor candidates. The majority really were "yes" or "no."

That experience, incidently, is one reason I view with deep skepticism the claims by many of the slashdot crowd that there's plenty of hotshot American programmers looking for jobs, so we don't need no steekeen unwashed H1-B visa holders. Many if not most of the ones we didn't call back were the sort who, for instance, claimed years of MySQL, Oracle, and Sybase experience but couldn't tell me about any differences in their SQL dialects. Or the Java programmers who got wrong something like

class Dog {
    public String name;
}

static void foo(Dog d) {
    d = new Dog();
    d.name = "Rex";
}

...

Dog d = new Dog();
d.name = "Prince";
foo(d);

// What is d's name?

So while contract trial periods can be useful, I think there are solid reasons why you don't see companies doing that more often. If you really want to move into the hot new *cough* Python field, work on an open-source project. Start a blog. Bring a USB drive with code you've written to the interview. Floss. Don't be a "maybe."

Wednesday, May 11, 2005

Materialized views in PostgreSQL

One thing I've wanted to write about for a while is materialized views in PostgreSQL. Materialized views are basically precomputed views; they're very very useful if you have an expensive query against data that doesn't change much. This is one of the tricks I use to keep the db overhead on Carnage Blender reasonable.

Oracle and DB2 are the only RDBMS products that can auto-materialize views for you, but you can rig them yourself in PostgreSQL or anything that gives you a powerful enough trigger + procedural language. (And if performance is crucial, you'll end up hand-optimizing them in Oracle/DB2 as well. At least, back when I used Oracle in the 8.1 days.)

But while looking for examples of a certain PL/PGSQL feature this morning, I found Jonathan Gardner's materialized views page (link may be down; see the archive.org copy) which covers the subject so well that now I don't have anything left to add. Nicely done!

Incidently, this is just one more reason real procedural languages in the database are critical for most substantial projects. Implementing materialized views in your application logic would be worse than a joke.

Monday, May 09, 2005

Emacs support for Spyce

Committed spyce.el in svn contrib/. (For those of you not bound to it by 10+ years of habit, el stands for Emacs Lisp.) I didn't write a "spyce mode" per se; I just used the mmm-mode module to allow Emacs to recognize the python snippets inside html. mmm-mode is one of the most complex modules I've had to work with, and while it gets the job done, to me it seems like emacs is showing its age. Maybe one of the DrPython guys will write an Emacs keybinding package so I don't feel too lost.

This is about as simple as you get with mmm-mode. You can get a lot more complicated.

(require 'mmm-auto)
(require 'mmm-sample)
(setq mmm-global-mode 'maybe)
(setq auto-mode-alist (cons '("\\.spy$" . html-mode) auto-mode-alist))
(setq auto-mode-alist (cons '("\\.spi$" . html-mode) auto-mode-alist))

(mmm-add-classes
 '((spyce1
    :submode python-mode
    :front "<%"
    :back "%>")
   (spyce2
    :submode python-mode
    :front "\\[\\["
    :back "\\]\\]")
  )
)
(mmm-add-mode-ext-class nil "\\.spi$" 'spyce1)
(mmm-add-mode-ext-class nil "\\.spy$" 'spyce1)
(mmm-add-mode-ext-class nil "\\.spi$" 'spyce2)
(mmm-add-mode-ext-class nil "\\.spy$" 'spyce2)

Thursday, May 05, 2005

Looks like spyce will have to find a new home

SF.net just sent out an email to project admins saying, among other things, that project web space will soon be mounted read-only. Bleh. That means I just wasted a LOT of time tweaking the Spyce makefile to auto-deploy to sf.net for no reason, because both of the new demos want to be able to write to disk. (One because it uses durus, and the other because since SF runs in CGI mode, the pool module pickles and unpickles itself to disk for each request. Which sucks rocks for performance, of course, but if you cared about that you wouldn't be using CGI, now would you?)

Of course spyce.org is already taken by someone who isn't using it...

I guess the bright side is now I can go ahead and enable comments on the Spyce documentation once it moves, if I have the time to wrestle with the TOC (table-of-contents) module to get it to spit out appropriate .spy files instead of html. (Making it into a wiki is another attractive option, but the nice thing about the TOC module is you can easily generate a single-page version for printing.)

(Incidently, with SF apparently modernizing many of the other aspects of their operation, when are they going to offer a better database than MySQL?)

"a new web framework named spyce"

Dion Almaer writes in a sort of footnote to his Python has a leg up in the dynamic language race post that

[T]here is also word that a new web framework named spyce is going to try to do what Rails has done for Ruby.

I think there are definitely some points of commonality if you take a big picture view; primarily, both are about leveraging the capabilities of dynamic languages to provide an environment vastly more productive than is possible with traditional staticly-typed-languages (Java, C#).

How we go about that, though, is quite different. Spyce isn't trying to be a Rails clone a la Subway; it's its own beast. Spyce is more page-centric, like ASP.NET or OpenACS. Like all modern frameworks, Spyce encourages separation of code and markup, but it's not religious about forcing one man's vision of MVC down your throat.

And while strictly speaking Dion is wrong to call Spyce "new," Spyce 2.0 has a much, much more modern feel to it than 1.x. If you evaluated Spyce a couple years ago, have another look.

Wednesday, May 04, 2005

Spyce 2.0 beta released

Some 217 commits into development, Spyce 2.0 is finally available for download. The main Spyce site has been updated with the new docs, examples, and demos.

Full changelog is here; highlights include

  new-style active tags 
    - docs
  OpenACS-like (Tiles-ish, for you JSP people) parent/child templating system
    - roughly the same speed as include.spyce, so using one parent template
      has about 1/2 the overhead as the old standard 
      two-includes-for-header-and-footer.
    - docs
  Active Handlers
    reusable components without the leaky abstraction of ASP.NET et al.
    - docs

The only major change since the prerelease was announced on this blog is that the class chunk token was changed from [[\\ to [[!. At least that's the only one I remember after spending the last 5 days wrestling with sourceforge's cgi-bin.

Friday, April 29, 2005

how well do you know python, part 6

class foo(list):
    def __eq__(self, other):
        raise 'foo.__eq__ called'

>>> help(list.__eq__)
Help on wrapper_descriptor:

__eq__(...)
    x.__eq__(y) <==> x==y

>>> [].__eq__(foo())
True
>>> [] == foo()
Traceback (most recent call last):
  File "", line 1, in ?
  File "", line 3, in __eq__
foo.__eq__ called

Help says those two statements should be equivalent. Why aren't they?

Wednesday, April 27, 2005

Any questions?

Closing on a real beta for Spyce 2.0, is there anything you'd like to see better illustrated? I have time to do another demo in the next couple days provided the scope is reasonable.

Tuesday, April 26, 2005

Extending Spyce with spyceModule

One of the ways to extend Spyce is with a "Spyce module." This is a historical term, and a little unfortunate because some people have assumed when I talk about "modules" I automatically mean the Spyce variety rather than the vanilla python variety.

A Spyce module is simply a class that extends spyceModule.spyceModule. That's it.

Spyce modules may be used in a .spy page with

[[.import name="modulename"]]

which instructs the compiler to create code that instantiates an instance of the given spyceModule at the beginning of each request for this page. It also automatically invokes the instance's finish method when the request finishes.

What can you do with Spyce modules? A common reason to write a Spyce module -- perhaps the most common -- is is to provide some sort of resource pooling. Here's the spydurus module which does exactly that:

spydurus.py

CONNECTIONS = 3 # max connections to put in the pool

import Queue
from spyceModule import spyceModule

from durus.client_storage import ClientStorage
from durus.connection import Connection

q = Queue.Queue()
for i in range(CONNECTIONS):
    q.put(Connection(ClientStorage()))

class spydurus(spyceModule):
    def start(self):
        self.conn = None
        try:
            self.conn = q.get(timeout=10)
        except Queue.Empty:
            raise 'timeout while getting durus connection'
        else:
            self.conn.abort() # syncs connection

    def finish(self, err):
        q.put(self.conn)

That's it! (Well, minus some convenience methods dealing with self.conn that are elided for concision here.) Note that finish is always called, even if your code raises an exception, even if you redirect to another page, even if another module's finish method raises.

Most modules can also be usefully used as a python module -- for instance, tododata.py in the Spyce to-do demo uses spydurus.q to get one of the pooled connections in a non-.spy context.

spydurus doesn't need it, but spyceModule also provides a hook into the spyce server in the form of self._api. You can do just about anything from this. I've seen a module that implements cgi_buffer functionality, a module that provides a Struts-style controller, and more. And of course there's all the standard modules.

(Updated links to point to the sf.net site.)

Monday, April 25, 2005

how well do you know python, part 5

I was reminded of this one while working on the Spyce/Durus demo to make it a better example of using Durus in a real application.

What is wrong with the following code?

A.py:

import time, threading

finished = False

def foo():
    import sys
    sys.stderr.write('testing')
    global finished
    finished = True

threading.Thread(target=foo).start()
while not finished:
    time.sleep(1)

B.py:

import A

print 'finished'

Update: fixed A.py so there was only one problem.

Thursday, April 21, 2005

Spyce 2.0 prerelease

There won't be an official beta this week after all; Rimon wants to review the code and docs over the weekend before we take that step.

However, as an unofficial beta, Spyce 2.0 is available from subversion:

svn co http://svn-hosting.com/svn/spyce/trunk/spyce

The featureset is solid at this point. If you start playing around now, you won't get burned by changes of that sort. You may well find bugs, of course, which I'll fix as soon as I can.

The docs are up here, including a pretty good introduction as to Why Spyce 2.0 Rocks, if I say so myself. (That is, I say the intro is pretty good. But also that Spyce2 rocks, for that matter.) The to-do demo and an enhanced version of the chatbox demo are both included.

Update: forgot a link to the changelog.

Not your father's spyce

You've taken spyce off my "disgusting template engines" list. This is how I'd want to see apps structured, exactly.
    -- Tim Lesher

As a followup to my Durus notes, I've put up a new Spyce + Durus demo showing off the features of Spyce 2.0.

This time I allowed myself more than two files, and split the action logic into a separate module. (To-do lists seem to be the new standard for demoing your web toolkit, and I dare not be less than trendy.)

I tried to comment the code (linked from the demo pages) so it speaks for itself. If you have questions after looking at that and the language constructs page, please feel free to ask in the comments here. I'd like the docs to be in good shape for the 2.0 release.

Update: I've modified the demo to include spydurus, a Durus connection-pooling module for Spyce, and to use ClientStorage instead of FileStorage. The demo was popular enough that single-threading it got to be annoying.

(Updated links to point to the spyce sf.net site.)

Wednesday, April 20, 2005

Introduction to Durus

The README that comes with Durus is missing a couple pieces of information that are critical if you actually want to write a program that uses Durus, and the only other documentation appears to be the 2005 Durus pycon presentation, which gives an admirable description of the technical underpinnings but doesn't fill in the blanks of how to use it, either.

Specifically, as far as code samples go, the README gives you this:

Example using FileStorage to open a Connection to a file:

    from durus.file_storage import FileStorage
    from durus.connection import Connection
    connection = Connection(FileStorage("test.durus"))

And this:

    # Assume mymodule defines A as a subclass of Persistent.
    from mymodule import A 
    x = A()
    root = connection.get_root() # connection set as shown above.
    root["sample"] = x           # root is dict-like
    connection.commit()          # Now x is stored.

That's all you get.

Okay, in this situation it's clear how to retrieve x -- look it up by the key we've hardcoded. But what if x is a list of Persistent objects, and I want to retrieve one of those. Do I have to keep track of its index in x? Do I need to generate my own unique IDs? (Note that the linked code has an obvious race condition in a multithreaded environment.)

No, you don't. Durus assigns each Persisent object an attribute called _p_oid. (P object id. Not sure if P stands for Persistent, or Private, or something else entirely.)

[in persistent.py]
    def __new__(klass, *args, **kwargs):
        [...]
        instance._p_oid = None # <-- this is the oid that get() cares about

    def _p_format_oid(self):
        return format_oid(self._p_oid)

_p_oid is a four-byte binary string, so for passing around as a GET or POST variable in a web application, the format function (which turns it into a string representation of the oid number) is handy.

Now that you're passing oids around, Durus gives you an easy way to retrieve the object it identifies:

[in connection.py]
    def get(self, oid):
        """(oid:str|int|long) -> Persistent | None
        Return object for `oid`.

        The object may be a ghost.
        """

Note that if you do pass around the formatted id, you'll need to turn it into a Python int (or long) before sending it to get; if you pass the string '123' get will assume it's a valid (binary) oid and not autoconvert it.

Now that the code diving is out of the way, I'm enjoying Durus a lot. Next post I'll give a short Spyce demo using Durus for persistence.

Monday, April 18, 2005

Spyce tag compilation example

I had the question,

What do you mean by "compiling" tag libraries?

I mean, that Spyce compiles that chatbox.spy into in a more-or-less 1.3-legal python module. (The classcode/handlers/exports features aren't in 1.3, but you get the idea.) The compiled result looks like this (output courtesy of the Spyce -c option):

class boxlet(spyceTagPlus):
  name='boxlet'
  buffer=False
  exports=1
  classcode=((7,4),(10,83),"def addLine(self):\n    # (use get() in case server restarted)\n    request._api.getServerGlobals().get('chatlines', []).append(request['newline'])",'test-chatbox.spy')

  handlers=[('e73068ffe2d1ead1793b6790ec48f92a04fd18641','self.addLine')]

  def syntax(self):
    self.syntaxSingleOnly()
  
  def begin(self,width='300',lines='5'):
    pool=self._api._startModule('pool',None,None)
    taglib=self._api.getModules()['taglib']
    taglib.load('form','form.py','f')
    self._out.writeStatic('\n\n')
    pool.init()
    self._out.writeStatic('\n<div width="')
    self._out.writeExpr(width)
    self._out.writeStatic('">\n')
    i=-int(lines)
    line=None# for first export

    for line in pool.setdefault('chatlines',[])[i:]:
      self._out.writeStatic('  <div>')
      self._out.writeExpr(line)
      self._out.writeStatic('</div>\n')
      
    self._out.writeStatic('  <div>\n  <f:text name=newline />')
    taglib.tagPush('f','text',locals(),{'name':'newline'},False)
    try:
      taglib.tagBegin()
      taglib.tagBody()
      taglib.tagEnd()
    finally:taglib.tagPop()

    self._out.writeStatic('\n  <f:submit handler=\'self.addLine\' value="Send" />')
    taglib.tagPush('f','submit',locals(),{'_handlerid':'e73068ffe2d1ead1793b6790ec48f92a04fd18641','value':'Send'},False)
    try:
      taglib.tagBegin()
      taglib.tagBody()
      taglib.tagEnd()
    finally:taglib.tagPop()

    self._out.writeStatic('\n  <f:submit value="Refresh" />')
    taglib.tagPush('f','submit',locals(),{'_handlerid':'e73068ffe2d1ead1793b6790ec48f92a04fd18642','value':'Refresh'},False)
    try:
      taglib.tagBegin()
      taglib.tagBody()
      taglib.tagEnd()
    finally:taglib.tagPop()
    self._out.writeStatic('\n  </div>\n</div>\n\n')
    self.line=line
  
  def export(self):
    return{'last':self.line}
    
class spyceTagcollection(spyceTagLibrary):
  tags=[boxlet]

You can see why not many of these got written. :)