Spyced

I'm losing patience with sqlite. I've been working on Spyce examples using postgresql, but now that I'm getting close to releasing Spyce 2.1, I figured I'd better convert the examples to use sqlite since that's such a no-brainer to set up. It has been a frustrating experience. Weird-assness I've run into includes "int" is not the same as "integer" (always use the latter to avoid unpleasant surprises) It's impossible to get useful information about column DEFAULTs programmatically There's a bug in the parsing of some three-way joins And I didn't think I was doing anything very complicated! My examples have three tables at most! Really my overall impression is one of a "0.9" product at best. I'm amazed that so many people appear to use this festering pile of gotchas in production.

Apparently PragDave had the temerity to suggest at RailsConf that Rails could stand some improvement in some areas, such as supporting composite primary keys in ActiveRecord. Naturally, the first reaction of a huge Rails fan like Martin Fowler is to get to work figuring out how to implement this. Whoops, sorry, no, that would be in some alternate universe where fanboyism isn't the most important technical prinicple for some people. Martin's real reaction was to write a rebuttal , the gist of which is, if Rails doesn't already support it, it can't be important. It's sad when someone who's done some good work puts on the fanboy blinders. You used to get the same schtick from mysql.com back in the 3.x days. Remember the rants in their docs about how foreign keys and transctions were for wimps and real programmers didn't want them anyway? Of course, today even mysql corporate admits that these are important features, even though one can be forgiven for ...

Andrew Smith pointed out that according to Indeed.com, Python is about a factor of 3 times more popular than Ruby and is maintaining that lead as both graphs trend upwards. I'd like to add just a couple things that I noticed. One is that, like Django , Rails is a term with multiple meanings, and the Ruby framework only accounts for a small fraction of jobs that Indeed pulls up for that term. (I'm impressed that Indeed allows you to nest arbitrarily complex boolean expressions here...) Another is that although Python looks pretty popular vs Ruby or Lisp, it's a good thing that popularity doesn't really reflect how good a language is, because ye olde statically compiled languages are still seven to twenty times more popular than python . Even PHP and Perl are more popular. (Although the trend on Perl is definitely down-sloping, for which we can all give thanks. C++ also has a noticable downward trend.)

I wrote a relatively simple multithreaded script to automate some cleanup work in my database. I used psycopg1, because it was conveniently packaged for the version of debian the server had. (And also because psycopg2's bundled pooling mechanism kind of sucks.) My script ran for a couple minutes, and segfaulted. I upgraded to the latest version of psycopg1, to no avail. You'd think that after 20+ "stable" releases this wouldn't be a problem anymore. Sigh. I ran it in gdb to see where it was segfaulting, and sure enough psycopg was dereferencing a null pointer. Unfortunately it was far from obvious how to fix the problem, at least to someone unfamiliar with the code. I bit the bullet and upgraded to psycopg2, which apparently got its first non-beta release earlier this month. For less-sucky pooling I used sqlalchemy's pool module. No more segfaults.

Greg Mullane has an excellent post on updating unique columns . A simple problem, but one that can be troublesome in practice: [T]here is one circumstance when [unique constraints] can be a real pain: swapping the values of the rows. In other words, a transaction in which the column values start unique, and end unique, but may not be so in the middle. Read his article -- I wouldn't have thought of his "reversing the polarity" method. Clever! But my first thought when I read this was, "Aha, Greg missed one." Surely the easiest way is to simply create a deferrable constraint (where you can elect to have the constraint only checked at the end of the transaction, instead of at the end of each statement)! So I gave it a try: => CREATE TABLE foo ( i int CONSTRAINT foo_pk PRIMARY KEY DEFERRABLE ); ERROR: misplaced DEFERRABLE clause At first I thought this indicated a syntax error, but my syntax was correct. After some g...

Python database tools have tended to suffer from the the 80% problem . (Open-source hackers tend to come up with solutions that solve 80% of a problem. Then someone else comes along and covers a different 80% of the same problem. And so on, so you end up with different solutions that attack the same problem, none of which are general enough for others to build on.) SQLAlchemy is making this a thing of the past, thanks to Mike Bayer's hard work. And, increasingly, others. SQLAlchemy made its second major release today, the big zero-dot-two-oh. (Mike is conservative with version numbers; most projects would call this 0.9 if not 1.0.) There's also a migration guide for porting 0.1-based code. SQLAlchemy lives up to its billing as The Python SQL Toolkit and Object Relational Mapper. (Emphasis mine.) This is possible is because of the extensive under-the-hood effort Mike has expended keeping dependencies to a minimum; you really can build on its functionality at any level...

My friend Paul introduced Noodle at the Utah python user group a week ago. (Yeah, sorry about the not-exactly-breaking-news.) Paul's a Lisp expert -- I think from before he was a Python expert, but I'm actually not sure of the chronology there -- and he wrote Noodle to create a pythonic Lisp dialect. Noodle combines Lisp syntax and features like macros with Python-ish syntax for lists, dicts, and tuples, and compiles to Python bytecode so it can easily leverage all the Python libraries. This bears repeating: there are a lot of projects out there that try to produce Java bytecode or CIL, but Noodle is the first I've heard of that produces Python bytecode. Pretty cool, if you ask me. His slides are linked from his blog, but basically his conclusion so far is that it turns out to be harder to integrate python-style syntax into Lisp than he'd hoped. Not hard as in implementation, but hard as in making it non-clunky to use. The warts are small small individually bu...

I presented on PyGame at the utah python user group last night. (When I don't get someone else lined up to speak in advance, I end up doing it myself. You'd think that would be enough motivation to not procrastinate.) I had a lot of fun preparing this. I'd never used PyGame before, but as a teenager I spent a lot of time in the same space. (Anyone remember YakIcons?) So the general concepts were familiar to me, and I was pleasantly surprised by how good a job PyGame did at making things easy for me. Here are my pygame slides , and my pyquest game skeleton is here . (PyQuest is of course inspired by Crystal Quest -- the mac game, not the XB 360 remake -- and the graphics and sound files are from Solar Wolf, which I guess makes PyQuest LGPL. This caused Paul Cannon some serious mental trauma at the meeting, seeing and hearing solar-wolf-and-yet-not-solar-wolf.)

[Update Oct 2006: here is another serving of SqlSoup . Update 2: SqlSoup documentation is now part of the SQLAlchemy wiki.] Ian Bicking wrote in Towards PHP that a successful Python PHP-killer (as Spyce aspires to be) will need to include a simple data-access tool that Just Works. I had two thoughts: He's absolutely right I could do this with SqlAlchemy in an afternoon My afternoons are in short supply these days, and it took two of them, counting the documentation. But it's live now, as sqlalchemy.ext.sqlsoup. (The 0.1.5 release includes a docless version of sqlsoup; I recommend the subversion code until Mike rolls a new official release.) SqlSoup is NOT an ORM per se, although naturally it plays nicely with SqlAlchemy. SqlSoup inspects your database and reflects its contents with no other work required; in particular, no model definitions are necessary. Here's what SqlSoup looks like, given pyweboff-ish tables of users, books, and loans (SQL to set up t...

I spent some time yesterday researching (free) database replication options. Judging from the newsgroup posts I saw, there's a lot of confusion out there. The most common use case appears to be failover, i.e., you want to minimize downtime in the face of software or hardware failure by replicating your data across multiple machines. But, the most commonly-used options are completely inappropriate for this purpose. As Josh Berkus explained, there are two "dimensions" to replication : synchronous vs async, and master/slave vs multimaster. For a failover solution, if you want database B to take over from database A in case of failure, with no data loss, only synchronous solutions make sense . By definition, asynchronous replication means that database A can commit a transaction before those changes are also committed to database B. If A happens to fail between commit and replication, you've lost data. If that's not acceptable for you, then neither is async ...

For our March meeting, the Utah python user group had multiple people present a solution to the PyWebOff challenge . This is not an attempt at an authoritative web frameworks review. The backgrounds of the presenters are too different -- in particular, only the Rails and Spyce presenters had prior prior experience with the frameworks they used. That said -- and obviously I'm all sorts of biased as both presenter and maintainer of Spyce -- I think Spyce came off looking pretty well. Partly because it was designed to automate repetitive tasks with relatively little "magic," and partly because Spyce doesn't try to put you in a straitjacket: you can write strict MVC code if you want, but if mixing some code into your view makes more sense than the alternatives, you can do that too. "I," "me," etc. in the remainder of this post refers to the presenter speaking of himself. In presentation order, here are my notes: TurboGears Presenter: Byron C...

Mike Orr's pycon 2006 writeup is out in the March Linux Gazette . The usual suspects are present (keynote summaries, etc) as well as some lightning talk info that I haven't seen blogged elsewhere. (I loved Chad Whitacre's Testosterone screencast: "The Manly Python Testing Interface.")

I presented an IDE review at PyCon last Friday. It was basically a re-review of what I thought were the 3 most promising IDEs from the Utah Python User Group IDE review , to which I added SPE, which was by far the most popular of the ones we left out that time. The versions reviewed are: PyDev 1.0.2 SPE 0.8.2.a Komodo 3.5.2 Wing IDE 2.1 beta 1 I'd intended to base my presentation around a comparison of writing a smallish program in each of the IDEs, but the more I tried to make this not suck, the more I realized it was a losing proposition. Instead, I decided to try to focus on the features in each that most set them apart from the others (both positive and negative); this seemed more likely be useful. (I did a new feature matrix for this review, which is included after my comments. The slides I used are also up, at http://utahpython.org/jellis/pycon-ides.pdf , but aren't very useful absent video of the presentation itself. Hence this post.) PyDev PyDev has g...

Earlier, I wrote about how ORM developers shouldn't try to re-invent SQL . It doesn't need to be done, and you're not likely to end up with an actual improvement. SQL may be designed by committee, but it's also been refined from thousands if not millions of man-years of database experience. The same applies to DDL. (Data Definition Langage -- the part of the SQL standard that deals with CREATE and ALTER.) Unfortunately, a number of Python ORMs are trying to replace DDL with a homegrown Python API. This is a Bad Thing. There are at least four reasons why: Standards compliance Completeness Maintainability Beauty Standards compliance SQL DDL is a standard. That means if you want something more sophisticated than Emacs, you can choose any of half a dozen modeling tools like ERwin or ER/Studio to generate and edit your DDL. The Python data definition APIs, by contrast, aren't even compatibile with other Python tools. You can't take a table definition ...

I presented at the utah python user group last night. I gave an introduction to the PyDO2 ORM tool as well as the python DB API and psycopg. Most readers are probably most interested in the pydo2 section. Here's the slides: http://utahpython.org/data/python-and-databases.pdf . I briefly mentioned SqlAlchemy in the context of, "here's some stuff that SqlAlchemy does that pretty much nobody else can," but didn't have time to cover TWO ORM tools. It's worth having a look at though if you've reached the limits of what pydo2 et al can do.

Take Alex Martelli's Number class from the augmented assignment example in What's New in Python 2.0 : class Number: def __init__(self, value): self.value = value def __iadd__(self, increment): return Number( self.value + increment) >>> n = Number(5) >>> n That's not very pretty. Let's add a __getattr__ method and leverage all those nice methods from the int or float or whatever it's initialized with: class Number: def __init__(self, value): self.value = value def __iadd__(self, increment): return Number( self.value + increment) def __getattr__(self, attr): return getattr(self.value, attr) >>> n = Number(5) >>> n 5 Great, the __str__ method from our int is being used. Let's keep going with the example now: >>> n += 3 >>> n.value Traceback (most recent call last): File " ", line 1, in ? AttributeError: 'int' object has no att...

Glyph Lefkowitz cites me as an inspiration to write Why Axiom Doesn't Expose SQL . Alas, I disagree with most of what he says. My post was about how if you're writing a tool that presupposes the use of a relational database, it's stupid to try to protect your users from having to know SQL . (This also means I think projects that bend over backwards to pretend ALTER TABLE is too hard are misguided, as well. But that's another subject.) Glyph's first argument is that any form of SQL is an invitation to sql injection attacks. This particular form of scare mongering isn't appreciated. Come on: this is 2005. It's ridiculously easy to write injection-proof SQL, even by hand. Arguing that allowing SQL allows injection attacts is like arguing that coding in python allows "shutil.rmtree('/')": correct, but irrelevant. Glyph further claims that "interfaces should be complete things," and that this justifies trying to obliterate any ...

Spyce user Lars Schmidt created a Wikipedia article on Spyce. Nice! (Now I need to add a section on Active Handlers , which is one of the main things I think makes Spyce compelling for larger projects, as well as simplifying smaller ones.)

If you are writing an ORM tool, please keep this one point in mind: I already know SQL. Your new query syntax isn't better; it's just different. Which means one more thing for me to learn. Which means I probably won't bother, and will use instead an ORM tool that respects my time. Consider: if your target audience is like me, and already knows SQL, this should go without saying. Resist the temptation to be "clever" and overcomplicate things. If your target audience does not know SQL, then either they intend to learn SQL, because they're using a relational database and expect that non-python code or ad-hoc queries will be necessary at some point, or they have no intention of learning SQL, and all their data-access code will now and forever be in python, in which case they would be better off using Durus or ZODB or another OODB. Be honest with these people and everyone will be happier. Django does some cool stuff, but heaven help me if I ever had...

Trying to figure out why sometimes disk access on my test machine takes way, way too long -- 1000+ ms -- I wrote some test code. My threads ran a function that looks like this: write = [] def writer(): while True: start = time.time() f = tempfile.TemporaryFile() f.write('a' * 4000) end = time.time() write.append(end - start) Compare the times for max(write) on a machine with a SATA disk and on one with parallel ATA, where the given number of threads are run for a 10 second period: threads pata sata 1 6ms 6ms 2 400ms 11ms 4 1300ms 24ms Ouch. I admit I'm not a hardware nerd. Quite possibly I'm missing something, because even PATA shouldn't be THAT bad. Right? hdparm -i says for the PATA disk: /dev/hda: Model=ST380011A, FwRev=8.01, SerialNo=4JV59KZT Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% } RawCHS=16383/16/63,...