Spyced

I don't blog about my day job at Mozy much, but I can't pass this one up: Walt Mossberg reviewed Mozy vs Carbonite in today's Wall Street Journal, and he concluded "of the two products, I prefer Mozy." "The Walt effect" makes the Digg effect look like small potatoes.

As of Komodo 4.0 beta 2 , Komodo no longer includes ActiveState's Tk-based GUI builder (with support for non-tcl Tk bindings like Python's Tkinter). They've released it as open source to the SpecTcl project, from which it was apparently forked long ago. Re-integration with Komodo 4 as a plugin is planned eventually, but it doesn't look likely before the 4.0 release. I wonder how much of the impetus behind this is the increased amount of web-based development done today, and how much is due to Tk's increasingly dated look and the increased popularity of more modern toolkits such wxwidgets.

I needed to scan some pretty massive gzipped text files, so my first try was the obvious "for line in gzip.open(...)." This worked but seemed way slower than expected. So I wrote "pyzcat" as a test and ran it against a file with 100k lines: #!/usr/bin/python import sys, gzip for fname in sys.argv[1:]: for line in gzip.open(fname): print line, Results: $ time zcat testzcat.gz > /dev/null real 0m0.329s $ time ./pyzcat testzcat.gz > /dev/null real 0m3.792s 10x slower -- ouch! Well, if zcat is so much better, let's try using zcat to do the reads for us: def gziplines(fname): from subprocess import Popen, PIPE f = Popen(['zcat', fname], stdout=PIPE) for line in f.stdout: yield line for fname in sys.argv[1:]: for line in gziplines(fname): print line, Results: $ time ./pyzcat2 testzcat.gz |wc real 0m0.750s So, reading from a zcat subprocess is 5x faster than using the gzip module. cGzipFile anyo...

Mark Ramm will be giving a talk on SQLAlchemy . I'll be giving a talk on SqlSoup , the SQLAlchemy extension I wrote, as well as a tutorial on Advanced Databases with SQLAlchemy. For my tutorial, I'll be targetting people who understand database fundamentals but want to learn about more advanced features like triggers and how an ORM like SQLAlchemy lets you take advantage of those. (Many ORM tools force you to give up the more powerful database features and pretend instead that your database is a dumb object store, which IMO defeats one of the main purposes of using a modern database.) If you need to brush up on fundamentals first, Steve Holden is running a more basic tutorial on databases with Python earlier in the day. Here's his outline ; his slides from pycon 06 on the same subject are also online. If there's something you'd like to see covered in my talk or tutorial, comments are welcome by email (jonathan at utahpython dot org) or right here.

This is interesting, because the conventional wisdom of idiots on slashdot continues to be "use postgresql if you need advanced features, but use mysql if all you care about is speed," despite all the head-to-head benchmarks I've seen by third parties showing PostgreSQL to be faster under load. (MySQL's own benchmarks, of course, tend to favor their own product. Go figure, huh.) Here's the latest, showing postgresql about 50% faster than mysql across a variety of hardware. But where MySQL really takes a pounding is when you add multiple cores / CPUs : MySQL adds 37% performance going from 1 to 4 cores; postgresql adds 226%. Ouch! (This would also explain why MySQL sucks so hard on the Niagra chip on the requests-per-second graph -- Sun sacrificed GHz to get more cores in.) As even low-end servers start to go multicore this is going to be increasingly important. Update: PostgreSQL core member Josh Berkus says : [This] is a validation of the last four y...

Another maintenance release. Changelog: db module requires PK definition up-front (instead of erroring out later) default login render tags give better feedback on unsuccessful login raise NameError instead of string exception for invalid eval'd tag attributes fixes for spyceProject fix kwargs render-through on spy: list and table tags fix auto-selecting of multiple defaults in compound controls fix for class-based exceptions in old-style tag exception handlers

Jeff Shell doesn't find Spyce tags easy to translate into "what HTML does this output?" That's my fault for writing crappy documentation , I guess, although I did think the examples helped a bit. :) So, briefly, here's how you translate the Spyce form tags: <f:text name=quest label="Question:" /> All the tags correspond closely with raw HTML. (This is part of Not Wasting Your Time.) If the HTML is "input type=foo," the corresponding Spyce tag is f:foo. So this will generate "input type=text." Name attribute is the same as in HTML; by default Spyce also sets the ID attribute to the same as name, for convenience working with javascript. ID may of course be overridden separately. The "Label" attribute generates a "label for=..." html tag pair. For most form elements, the label will be placed before the input; the exceptions are radio and checkbox buttons. Spyce will add a "value" attribute ...

I saw something cool in #postgresql on freenode tonight. There is a bot in the channel named "rtfm_please;" he'll reply to messages with pages of the postgresql documentation on that subject. For instance, (9:03:51 PM) jbellis: constraint (9:03:51 PM) rtfm_please: For information about constraint (9:03:51 PM) rtfm_please: see http://www.postgresql.org/docs/current/static/ddl-partitioning.html (9:03:51 PM) rtfm_please: or http://www.postgresql.org/docs/current/static/infoschema-table-constraints.html (9:03:51 PM) rtfm_please: or http://www.postgresql.org/docs/current/static/sql-set-constraints.html Unfortunately it's not too bright: (9:03:48 PM) jbellis: add constraint (9:03:49 PM) rtfm_please: nothing found :-( Apparently it's not really a search, but a manually-maintained collection of links (which explains why it's virtually instantaneous). I was amused to see this entry: (9:23:36 PM) jbellis: mysql gotchas (9:23:37 PM) rtfm_please: For info...

Earlier this year I wrote an introduction to SqlSoup , the SQLAlchemy extension that leverages SQLAlchemy's excellent introspection, mapping, and sql construction to provide a database interface that is both simple and powerful. Here's what SqlSoup has added since then (continuing with the books/loans/users example tables from pyweboff ). Full SqlSoup documentation is on the SQLAlchemy wiki . Set operations The introduction covered updating and deleting rows that had been mapped to Python objects. You can also perform updates and deletes directly to the database. >>> db.loans.insert(book_id=book_id, user_name=user.name) MappedLoans(book_id=2,user_name='Bhargan Basepair',loan_date=None) >>> db.flush() >>> db.loans.delete(db.loans.c.book_id==2) >>> db.loans.update(db.loans.c.book_id==2, book_id=1) >>> db.loans.select_by(db.loans.c.book_id==1) [MappedLoans(book_id=1,user_name='Joe Student',loan_date=datetime....

I don't follow Ruby development too closely (most of the info on it is still in Japanese, after all), but the US RubyConf was held recently so there's been an unusual number of English posts on Ruby, among them David Pollack's The Impending Ruby Fracture. David's article seems to consist of these points: Matz is uninterested in adding static bondage & discipline features to Ruby (true, as far as I know) "Enterprise" users won't be satisfied without said features (more on this below) There are a lot of Ruby runtimes out there right now (the most interesting part of the article) Therefore some Enterprise will co-opt one of the runtimes to fork Ruby and add the B&D features (wtf?) Summarized this way it looks faintly ridiculous, and yet nobody over on the programming reddit has called this out. Maybe I'm taking excessive liberties with David's article, but I don't think I am. The possibility of forking is part of what makes open sou...

I'm pleased that I made it into the round of 1000 in Google's Codejam . The Python support was much appreciated! (Actually, I'd be curious what the breakdown was by language of the contestants and qualifiers. I have no real stats but from the 50+ submissions I glanced at, all C#, Java, and C++, I would guess that topcoder's "Python might be too slow" disclaimer scared a lot of people away from Python. Or, like VB.NET, it just isn't that popular with this group. The ignomy!) I solved the 250 pt problem (problem set 5) very quickly, which is good because I didn't end up solving the 750 pt one at all. The 250 pt problem was You are given a tuple (integer) f that describes a set of values from a function f. The x-th element (zero-indexed) is the value of f(x). The function is not convex at a particular x value if there exist y and z values such that y Here was my brute force solution: class point: def __init__(self, x=0, y=0): self....

If you work in the web development area, or even dabble in it as a hobbyist, sooner or later you're going to write code for a project that needs authentication. Probably sooner than later. For a feature that gets used so frequently, it's remarkable to me that nobody has really done this right. Here are some basic principles for a good solution: A minimum of customization to work out-of-the-box Gentle complexity slope when more sophisticated behavior is needed Play nice with others Don't try to solve world hunger The first two are, I hope, no-brainers. The second two bear more explanation. Play nice with others: not everyone wants to authenticate against a Users table in a relational database. (Fairly common alternatives are LDAP or Unix logins.) If you bake in assumptions like this too deep, it causes problems. It might be worth the problems if it were impossible to provide both generality and ease of use, but such is not the case. Don't try to solve wo...

Most frameworks today have pretty good support for code/markup reuse towards providing a common look to a site without the "include a header, include a footer" clunkers you used to see. (Spyce uses "parent tags," which are as elegant as any and more powerful than most.) But what I want to talk about today is code/markup reuse in the small, at a level that corresponds to functions or methods in Python. Most frameworks today still suck at this. Approaches vary, but the important thing they have in common is not letting you use the same techniques you use in the rest of the framework as well as generally being clunky. For instance, with Rails, you define functions which you can then use from your views/templates/presentation layer, but within those functions you're back in 1996, stringing HTML together manually. Something like def faq_html(question, answers) html = ''' <table class="faq"> <thead> <tr...

Traditional view-oriented frameworks (such as PHP, or Spyce 1.3) do not handle control logic well. Today I'll show how Spyce 2 solves this problem with "active handlers," and tomorrow I'll show how this lets Spyce provide much more elegant code-reuse tools than either other view-oriented frameworks or MVC frameworks like Turbogears and Django. Control logic is the code that decides what happens next in response to a user action. Say we have a simple form to create new to-do lists, like this: <form> <input type=text name=name value=""> <input type=submit> </form> In a traditional view-oriented framework (like Spyce 1.3), you would process this form with code that looks something like this: [[name = request['name'] db.todo_lists.insert(name=name) api.db.flush() ]] That wasn't too bad. (Especially since I cheated and used the modern Spyce db api, AKA SqlSoup from SQLAlchemy.) But even with a simple, one-element for...

The revamed Spyce website announces "Spyce will not waste your time." Most web frameworks today waste your time with busywork at least some of the time. This is unacceptable; boilerplate code is tedious to write, an obstacle to good maintenance, and a distraction from real productivity. All the development of Spyce 2.0 and 2.1 has been towards eliminating common sources of busywork in web development. Today I'll give a couple concrete examples of how this applies to creating forms for user input. Let's say you want to have a select box that remembers what the user's last selection was. Pretty much any form these days needs logic like this when giving feedback inline. Many frameworks make you write this code out by hand; if you've ever developed in one of these, you know how quickly this gets old. Here's what you'd have to write in Spyce 2.1, given a list of options named options : <f:select name="foo" data="options...

Just some bug fixes this time: fix support for threadless python builds fix using compiled tags within other tags fix f:textarea fix spyceUtil.extractValue for incomplete dict work-alikes fix session1 backwards compatibility

This morning I read about the new Amazone Elastic Compute Cloud service. It's basically a cluster of Xen VPSes, done right. At least that's what I thought until I actually tried to use it. Let's see: Sign up for AWS Sign up for S3 Create certs Download tools Export 3 EC2 environment variables Oh, hell. Tools are in java. Switch to windows box since I don't have the patience to figure out installing Java 1.5 on debian right now. Repeat steps 4-5 Export JAVA_HOME Run ec2-describe-images Exception in thread "main" java.lang.NoClassDefFoundError: com/amazon/aes/webservices/client/cmd/DescribeImages Looks like I get to try to fix the classpath for them. How retro-cool can you get? It's just like 1999! Bah. Next time, use Python, guys. update: : Apparently they didn't even bother testing on windows and their script was just plain broken. Way to go.

For the first time, codejam 2006 features Python as a language option. I don't follow topcoder closely enough to know if this is the first contest they've done with Python or not, but this smells of Google influence to me. I was about resigned to not bothering with this year's code jam, with my C# even rustier than last year , but now I'm totally stoked. Go Python! (Oh yeah, link to main code jam page .) Update: for the curious, topcoder added Python support about 10 days ago

Check out What's new : Login tags SQLAlchemy integration Validation in handlers More powerful form tags Improved handler integration Pauli Virtanen has contributed debian scripts, so we'll have a .deb to go with the rpm and sourceball releases pretty soon.

I have a pg database that just suffered through a multiple-hour vacuum of one of its tables. It was pretty painful. Autovacuum is configured to run every 12 hours, but for whatever reason it didn't see fit to vacuum this very busy table for substantially longer than that. (Not sure exactly how long. Over a week is my guess.) The problem is that one size does not fit all. On smaller tables, the default autovacuum_vacuum_scale_factor of 0.4 is just fine. On this table waiting that long is unacceptable.

I'm losing patience with sqlite. I've been working on Spyce examples using postgresql, but now that I'm getting close to releasing Spyce 2.1, I figured I'd better convert the examples to use sqlite since that's such a no-brainer to set up. It has been a frustrating experience. Weird-assness I've run into includes "int" is not the same as "integer" (always use the latter to avoid unpleasant surprises) It's impossible to get useful information about column DEFAULTs programmatically There's a bug in the parsing of some three-way joins And I didn't think I was doing anything very complicated! My examples have three tables at most! Really my overall impression is one of a "0.9" product at best. I'm amazed that so many people appear to use this festering pile of gotchas in production.

Apparently PragDave had the temerity to suggest at RailsConf that Rails could stand some improvement in some areas, such as supporting composite primary keys in ActiveRecord. Naturally, the first reaction of a huge Rails fan like Martin Fowler is to get to work figuring out how to implement this. Whoops, sorry, no, that would be in some alternate universe where fanboyism isn't the most important technical prinicple for some people. Martin's real reaction was to write a rebuttal , the gist of which is, if Rails doesn't already support it, it can't be important. It's sad when someone who's done some good work puts on the fanboy blinders. You used to get the same schtick from mysql.com back in the 3.x days. Remember the rants in their docs about how foreign keys and transctions were for wimps and real programmers didn't want them anyway? Of course, today even mysql corporate admits that these are important features, even though one can be forgiven for ...

Andrew Smith pointed out that according to Indeed.com, Python is about a factor of 3 times more popular than Ruby and is maintaining that lead as both graphs trend upwards. I'd like to add just a couple things that I noticed. One is that, like Django , Rails is a term with multiple meanings, and the Ruby framework only accounts for a small fraction of jobs that Indeed pulls up for that term. (I'm impressed that Indeed allows you to nest arbitrarily complex boolean expressions here...) Another is that although Python looks pretty popular vs Ruby or Lisp, it's a good thing that popularity doesn't really reflect how good a language is, because ye olde statically compiled languages are still seven to twenty times more popular than python . Even PHP and Perl are more popular. (Although the trend on Perl is definitely down-sloping, for which we can all give thanks. C++ also has a noticable downward trend.)

I wrote a relatively simple multithreaded script to automate some cleanup work in my database. I used psycopg1, because it was conveniently packaged for the version of debian the server had. (And also because psycopg2's bundled pooling mechanism kind of sucks.) My script ran for a couple minutes, and segfaulted. I upgraded to the latest version of psycopg1, to no avail. You'd think that after 20+ "stable" releases this wouldn't be a problem anymore. Sigh. I ran it in gdb to see where it was segfaulting, and sure enough psycopg was dereferencing a null pointer. Unfortunately it was far from obvious how to fix the problem, at least to someone unfamiliar with the code. I bit the bullet and upgraded to psycopg2, which apparently got its first non-beta release earlier this month. For less-sucky pooling I used sqlalchemy's pool module. No more segfaults.

Greg Mullane has an excellent post on updating unique columns . A simple problem, but one that can be troublesome in practice: [T]here is one circumstance when [unique constraints] can be a real pain: swapping the values of the rows. In other words, a transaction in which the column values start unique, and end unique, but may not be so in the middle. Read his article -- I wouldn't have thought of his "reversing the polarity" method. Clever! But my first thought when I read this was, "Aha, Greg missed one." Surely the easiest way is to simply create a deferrable constraint (where you can elect to have the constraint only checked at the end of the transaction, instead of at the end of each statement)! So I gave it a try: => CREATE TABLE foo ( i int CONSTRAINT foo_pk PRIMARY KEY DEFERRABLE ); ERROR: misplaced DEFERRABLE clause At first I thought this indicated a syntax error, but my syntax was correct. After some g...

Python database tools have tended to suffer from the the 80% problem . (Open-source hackers tend to come up with solutions that solve 80% of a problem. Then someone else comes along and covers a different 80% of the same problem. And so on, so you end up with different solutions that attack the same problem, none of which are general enough for others to build on.) SQLAlchemy is making this a thing of the past, thanks to Mike Bayer's hard work. And, increasingly, others. SQLAlchemy made its second major release today, the big zero-dot-two-oh. (Mike is conservative with version numbers; most projects would call this 0.9 if not 1.0.) There's also a migration guide for porting 0.1-based code. SQLAlchemy lives up to its billing as The Python SQL Toolkit and Object Relational Mapper. (Emphasis mine.) This is possible is because of the extensive under-the-hood effort Mike has expended keeping dependencies to a minimum; you really can build on its functionality at any level...

My friend Paul introduced Noodle at the Utah python user group a week ago. (Yeah, sorry about the not-exactly-breaking-news.) Paul's a Lisp expert -- I think from before he was a Python expert, but I'm actually not sure of the chronology there -- and he wrote Noodle to create a pythonic Lisp dialect. Noodle combines Lisp syntax and features like macros with Python-ish syntax for lists, dicts, and tuples, and compiles to Python bytecode so it can easily leverage all the Python libraries. This bears repeating: there are a lot of projects out there that try to produce Java bytecode or CIL, but Noodle is the first I've heard of that produces Python bytecode. Pretty cool, if you ask me. His slides are linked from his blog, but basically his conclusion so far is that it turns out to be harder to integrate python-style syntax into Lisp than he'd hoped. Not hard as in implementation, but hard as in making it non-clunky to use. The warts are small small individually bu...

I presented on PyGame at the utah python user group last night. (When I don't get someone else lined up to speak in advance, I end up doing it myself. You'd think that would be enough motivation to not procrastinate.) I had a lot of fun preparing this. I'd never used PyGame before, but as a teenager I spent a lot of time in the same space. (Anyone remember YakIcons?) So the general concepts were familiar to me, and I was pleasantly surprised by how good a job PyGame did at making things easy for me. Here are my pygame slides , and my pyquest game skeleton is here . (PyQuest is of course inspired by Crystal Quest -- the mac game, not the XB 360 remake -- and the graphics and sound files are from Solar Wolf, which I guess makes PyQuest LGPL. This caused Paul Cannon some serious mental trauma at the meeting, seeing and hearing solar-wolf-and-yet-not-solar-wolf.)

[Update Oct 2006: here is another serving of SqlSoup . Update 2: SqlSoup documentation is now part of the SQLAlchemy wiki.] Ian Bicking wrote in Towards PHP that a successful Python PHP-killer (as Spyce aspires to be) will need to include a simple data-access tool that Just Works. I had two thoughts: He's absolutely right I could do this with SqlAlchemy in an afternoon My afternoons are in short supply these days, and it took two of them, counting the documentation. But it's live now, as sqlalchemy.ext.sqlsoup. (The 0.1.5 release includes a docless version of sqlsoup; I recommend the subversion code until Mike rolls a new official release.) SqlSoup is NOT an ORM per se, although naturally it plays nicely with SqlAlchemy. SqlSoup inspects your database and reflects its contents with no other work required; in particular, no model definitions are necessary. Here's what SqlSoup looks like, given pyweboff-ish tables of users, books, and loans (SQL to set up t...

I spent some time yesterday researching (free) database replication options. Judging from the newsgroup posts I saw, there's a lot of confusion out there. The most common use case appears to be failover, i.e., you want to minimize downtime in the face of software or hardware failure by replicating your data across multiple machines. But, the most commonly-used options are completely inappropriate for this purpose. As Josh Berkus explained, there are two "dimensions" to replication : synchronous vs async, and master/slave vs multimaster. For a failover solution, if you want database B to take over from database A in case of failure, with no data loss, only synchronous solutions make sense . By definition, asynchronous replication means that database A can commit a transaction before those changes are also committed to database B. If A happens to fail between commit and replication, you've lost data. If that's not acceptable for you, then neither is async ...

For our March meeting, the Utah python user group had multiple people present a solution to the PyWebOff challenge . This is not an attempt at an authoritative web frameworks review. The backgrounds of the presenters are too different -- in particular, only the Rails and Spyce presenters had prior prior experience with the frameworks they used. That said -- and obviously I'm all sorts of biased as both presenter and maintainer of Spyce -- I think Spyce came off looking pretty well. Partly because it was designed to automate repetitive tasks with relatively little "magic," and partly because Spyce doesn't try to put you in a straitjacket: you can write strict MVC code if you want, but if mixing some code into your view makes more sense than the alternatives, you can do that too. "I," "me," etc. in the remainder of this post refers to the presenter speaking of himself. In presentation order, here are my notes: TurboGears Presenter: Byron C...

Mike Orr's pycon 2006 writeup is out in the March Linux Gazette . The usual suspects are present (keynote summaries, etc) as well as some lightning talk info that I haven't seen blogged elsewhere. (I loved Chad Whitacre's Testosterone screencast: "The Manly Python Testing Interface.")

I presented an IDE review at PyCon last Friday. It was basically a re-review of what I thought were the 3 most promising IDEs from the Utah Python User Group IDE review , to which I added SPE, which was by far the most popular of the ones we left out that time. The versions reviewed are: PyDev 1.0.2 SPE 0.8.2.a Komodo 3.5.2 Wing IDE 2.1 beta 1 I'd intended to base my presentation around a comparison of writing a smallish program in each of the IDEs, but the more I tried to make this not suck, the more I realized it was a losing proposition. Instead, I decided to try to focus on the features in each that most set them apart from the others (both positive and negative); this seemed more likely be useful. (I did a new feature matrix for this review, which is included after my comments. The slides I used are also up, at http://utahpython.org/jellis/pycon-ides.pdf , but aren't very useful absent video of the presentation itself. Hence this post.) PyDev PyDev has g...

Earlier, I wrote about how ORM developers shouldn't try to re-invent SQL . It doesn't need to be done, and you're not likely to end up with an actual improvement. SQL may be designed by committee, but it's also been refined from thousands if not millions of man-years of database experience. The same applies to DDL. (Data Definition Langage -- the part of the SQL standard that deals with CREATE and ALTER.) Unfortunately, a number of Python ORMs are trying to replace DDL with a homegrown Python API. This is a Bad Thing. There are at least four reasons why: Standards compliance Completeness Maintainability Beauty Standards compliance SQL DDL is a standard. That means if you want something more sophisticated than Emacs, you can choose any of half a dozen modeling tools like ERwin or ER/Studio to generate and edit your DDL. The Python data definition APIs, by contrast, aren't even compatibile with other Python tools. You can't take a table definition ...

I presented at the utah python user group last night. I gave an introduction to the PyDO2 ORM tool as well as the python DB API and psycopg. Most readers are probably most interested in the pydo2 section. Here's the slides: http://utahpython.org/data/python-and-databases.pdf . I briefly mentioned SqlAlchemy in the context of, "here's some stuff that SqlAlchemy does that pretty much nobody else can," but didn't have time to cover TWO ORM tools. It's worth having a look at though if you've reached the limits of what pydo2 et al can do.

Take Alex Martelli's Number class from the augmented assignment example in What's New in Python 2.0 : class Number: def __init__(self, value): self.value = value def __iadd__(self, increment): return Number( self.value + increment) >>> n = Number(5) >>> n That's not very pretty. Let's add a __getattr__ method and leverage all those nice methods from the int or float or whatever it's initialized with: class Number: def __init__(self, value): self.value = value def __iadd__(self, increment): return Number( self.value + increment) def __getattr__(self, attr): return getattr(self.value, attr) >>> n = Number(5) >>> n 5 Great, the __str__ method from our int is being used. Let's keep going with the example now: >>> n += 3 >>> n.value Traceback (most recent call last): File " ", line 1, in ? AttributeError: 'int' object has no att...

Glyph Lefkowitz cites me as an inspiration to write Why Axiom Doesn't Expose SQL . Alas, I disagree with most of what he says. My post was about how if you're writing a tool that presupposes the use of a relational database, it's stupid to try to protect your users from having to know SQL . (This also means I think projects that bend over backwards to pretend ALTER TABLE is too hard are misguided, as well. But that's another subject.) Glyph's first argument is that any form of SQL is an invitation to sql injection attacks. This particular form of scare mongering isn't appreciated. Come on: this is 2005. It's ridiculously easy to write injection-proof SQL, even by hand. Arguing that allowing SQL allows injection attacts is like arguing that coding in python allows "shutil.rmtree('/')": correct, but irrelevant. Glyph further claims that "interfaces should be complete things," and that this justifies trying to obliterate any ...