Spyced

ActiveState has released Komodo IDE 4 . Perhaps more interesting, if you're not already a Komodo user, is the release of Komodo Edit , which is very similar to the old Komodo IDE Personal edition, only instead of costing around $30, Komodo Edit is free. The mental difference between "free" and "$30" is much more than the relatively small amount of money; it will be interesting to see what happens in the IDE space now. After a brief evaluation I would say Edit is perhaps the strongest contender for "best free python IDE." The only serious alternative is PyDev, which on its Eclipse foundation provides features like svn integration that Edit doesn't. PyDev also includes a debugger, another feature ActiveState would like to see you upgrade to the full IDE for. But Komodo is stronger in other areas such as call tips and, well, not being based on Eclipse. I also think its code completion is better, although this impression is preliminary. It...

See you there!

I was pretty happy with the old version of blogger, but I upgraded today so I can use the new API against my own blog. So far I have 4 spam comments (captcha is still on) versus about that number for the entire life of my blog under the old blogger. Bleh. Could just be a coincidence. I hope so. (Update Feb 26: A month later, I've had just one more spam comment. So it probably really was just coincidence.)

In December, Fujitsu made available a video of Gavin Sherry speaking on Advanced PostgreSQL . (Where's part 2, guys?) Here's some of the topics Gavin addresses, and the approximate point at which they can be found in the video. [start] wal_buffers: "at least 64"; when it's ok to turn fsync off [not very often]; how hard disk rpm limits write-based transaction rate, even with WAL 00:12: wal_sync_method = fdatasync is worth checking out on Linux 00:13: FSM [free space map], MVCC, and vacuum; how to determine appropriate FSM size; why this is important to avoid VACUUM FULL 00:22: vaccum_cost_delay 00:26: background writer 00:30: history of buffer replacement strategies 00:37: scenarios where bgwriter is not useful 00:41: how random_page_cost affects planner's use of indexes 00:47: effective_cache_size 00:49: logging; how to configure syslog to not hose your performance 00:52: linux file system configuration 00:58: solaris fs...

One of the reasons ORM tools have a spotted reputation is that it's really, really easy to write a dumb ORM that works fine for simple queries but performs like molasses once you start throwing real data at it. Let me give an example of a situation where, to my knowledge, only SQLAlchemy of the Python (or Ruby) ORMs is really able to handle things elegantly, without gross hacks like "piggy backing." Often you'll see a one-to-many relationship where you're not always interested in all of the -many side. For instance, you might have a users table, each associated with many orders. In SA you'd first define the Table objects, then create a mapper that's responsible for doing The Right Thing when you write "user.orders." (I'm skipping connecting to the database for the sake of brevity, but that's pretty simple. I'm also avoiding specifying columns for the Tables by assuming they're in the database already and telling SA to a...

Vadim Tkachenko posted an interesting benchmark of MyISAM vs InnoDB vs Falcon datatypes. (Falcon is the new backend that MySQL started developing after Oracle bought InnoDB.) For me the interesting part is not the part with the alpha code -- Falcon is competitive for some queries but gets absolutely crushed on others -- but how InnoDB is around 30% faster than MyISAM. And these are pure selects, supposedly where MyISAM is best. Of course this is a small benchmark and YMMV, but this is encouraging to me because it suggests that if I ever have to use MySQL, I can use a backend with transactions, real foreign key support, etc., without sucking too badly performance-wise. (It also suggests that people who responded to the post on postgresql crushing mysql in a different benchmark by saying, "well, if they wanted speed they should have used MyISAM," might want to reconsider their advice.)

I thought I was pretty used to SQL's three-valued logic by now, but this still caused me a minute of scratching my head: # select count(*) from _t; count ------- 1306 (1 row) # select count(*) from _t2; count ------- 19497 (1 row) Both _t and _t2 are temporary tables of a single column I created with SELECT DISTINCT. # select count(*) from _t where userhash in (select userhash from _t2); count ------- 982 (1 row) # select count(*) from _t where userhash not in (select userhash from _t2); count ------- 0 (1 row) Hmm, 982 + 0 != 1306... Turns out there was a null in _t2; X in {set containing null} evaluates to null, not false, and negating null still gives null. (The rule of thumb is, any operation on null is still null.) ................. I'm giving a tutorial on Advanced Databases with SQLAlchemy at PyCon in February. Feel free to let me know if there is anything you'd like me to cover specifically.

My thinkpad R52's screen died a couple days ago. I decided that this time I was going to be a man and install Linux on my new machine: all our servers run Debian, and "apt-get install" is just so convenient vs manual package installation on Windows. And it looks like qemu is a good enough "poor man's vmware" that I could still test stuff in IE when necessary. Alas, it was not to be. My new laptop is an HP dv9005, and although ubuntu's livecd mode ran fine, when it actually installed itself to the HDD and loaded X it did strange and colorful things to the LCD. Things that didn't resemble an actual desktop. When I told it to start in recovery mode instead it didn't even finish booting. That was all the time I had to screw around, so I reinstalled Windows to start getting work done again. Which brings me (finally!) to this advice on tortoisesvn : it really puts teh snappy back in the tortoise. Thanks annonymous progblogger!

I don't blog about my day job at Mozy much, but I can't pass this one up: Walt Mossberg reviewed Mozy vs Carbonite in today's Wall Street Journal, and he concluded "of the two products, I prefer Mozy." "The Walt effect" makes the Digg effect look like small potatoes.

As of Komodo 4.0 beta 2 , Komodo no longer includes ActiveState's Tk-based GUI builder (with support for non-tcl Tk bindings like Python's Tkinter). They've released it as open source to the SpecTcl project, from which it was apparently forked long ago. Re-integration with Komodo 4 as a plugin is planned eventually, but it doesn't look likely before the 4.0 release. I wonder how much of the impetus behind this is the increased amount of web-based development done today, and how much is due to Tk's increasingly dated look and the increased popularity of more modern toolkits such wxwidgets.

I needed to scan some pretty massive gzipped text files, so my first try was the obvious "for line in gzip.open(...)." This worked but seemed way slower than expected. So I wrote "pyzcat" as a test and ran it against a file with 100k lines: #!/usr/bin/python import sys, gzip for fname in sys.argv[1:]: for line in gzip.open(fname): print line, Results: $ time zcat testzcat.gz > /dev/null real 0m0.329s $ time ./pyzcat testzcat.gz > /dev/null real 0m3.792s 10x slower -- ouch! Well, if zcat is so much better, let's try using zcat to do the reads for us: def gziplines(fname): from subprocess import Popen, PIPE f = Popen(['zcat', fname], stdout=PIPE) for line in f.stdout: yield line for fname in sys.argv[1:]: for line in gziplines(fname): print line, Results: $ time ./pyzcat2 testzcat.gz |wc real 0m0.750s So, reading from a zcat subprocess is 5x faster than using the gzip module. cGzipFile anyo...

Mark Ramm will be giving a talk on SQLAlchemy . I'll be giving a talk on SqlSoup , the SQLAlchemy extension I wrote, as well as a tutorial on Advanced Databases with SQLAlchemy. For my tutorial, I'll be targetting people who understand database fundamentals but want to learn about more advanced features like triggers and how an ORM like SQLAlchemy lets you take advantage of those. (Many ORM tools force you to give up the more powerful database features and pretend instead that your database is a dumb object store, which IMO defeats one of the main purposes of using a modern database.) If you need to brush up on fundamentals first, Steve Holden is running a more basic tutorial on databases with Python earlier in the day. Here's his outline ; his slides from pycon 06 on the same subject are also online. If there's something you'd like to see covered in my talk or tutorial, comments are welcome by email (jonathan at utahpython dot org) or right here.

This is interesting, because the conventional wisdom of idiots on slashdot continues to be "use postgresql if you need advanced features, but use mysql if all you care about is speed," despite all the head-to-head benchmarks I've seen by third parties showing PostgreSQL to be faster under load. (MySQL's own benchmarks, of course, tend to favor their own product. Go figure, huh.) Here's the latest, showing postgresql about 50% faster than mysql across a variety of hardware. But where MySQL really takes a pounding is when you add multiple cores / CPUs : MySQL adds 37% performance going from 1 to 4 cores; postgresql adds 226%. Ouch! (This would also explain why MySQL sucks so hard on the Niagra chip on the requests-per-second graph -- Sun sacrificed GHz to get more cores in.) As even low-end servers start to go multicore this is going to be increasingly important. Update: PostgreSQL core member Josh Berkus says : [This] is a validation of the last four y...

Another maintenance release. Changelog: db module requires PK definition up-front (instead of erroring out later) default login render tags give better feedback on unsuccessful login raise NameError instead of string exception for invalid eval'd tag attributes fixes for spyceProject fix kwargs render-through on spy: list and table tags fix auto-selecting of multiple defaults in compound controls fix for class-based exceptions in old-style tag exception handlers

Jeff Shell doesn't find Spyce tags easy to translate into "what HTML does this output?" That's my fault for writing crappy documentation , I guess, although I did think the examples helped a bit. :) So, briefly, here's how you translate the Spyce form tags: <f:text name=quest label="Question:" /> All the tags correspond closely with raw HTML. (This is part of Not Wasting Your Time.) If the HTML is "input type=foo," the corresponding Spyce tag is f:foo. So this will generate "input type=text." Name attribute is the same as in HTML; by default Spyce also sets the ID attribute to the same as name, for convenience working with javascript. ID may of course be overridden separately. The "Label" attribute generates a "label for=..." html tag pair. For most form elements, the label will be placed before the input; the exceptions are radio and checkbox buttons. Spyce will add a "value" attribute ...

I saw something cool in #postgresql on freenode tonight. There is a bot in the channel named "rtfm_please;" he'll reply to messages with pages of the postgresql documentation on that subject. For instance, (9:03:51 PM) jbellis: constraint (9:03:51 PM) rtfm_please: For information about constraint (9:03:51 PM) rtfm_please: see http://www.postgresql.org/docs/current/static/ddl-partitioning.html (9:03:51 PM) rtfm_please: or http://www.postgresql.org/docs/current/static/infoschema-table-constraints.html (9:03:51 PM) rtfm_please: or http://www.postgresql.org/docs/current/static/sql-set-constraints.html Unfortunately it's not too bright: (9:03:48 PM) jbellis: add constraint (9:03:49 PM) rtfm_please: nothing found :-( Apparently it's not really a search, but a manually-maintained collection of links (which explains why it's virtually instantaneous). I was amused to see this entry: (9:23:36 PM) jbellis: mysql gotchas (9:23:37 PM) rtfm_please: For info...

Earlier this year I wrote an introduction to SqlSoup , the SQLAlchemy extension that leverages SQLAlchemy's excellent introspection, mapping, and sql construction to provide a database interface that is both simple and powerful. Here's what SqlSoup has added since then (continuing with the books/loans/users example tables from pyweboff ). Full SqlSoup documentation is on the SQLAlchemy wiki . Set operations The introduction covered updating and deleting rows that had been mapped to Python objects. You can also perform updates and deletes directly to the database. >>> db.loans.insert(book_id=book_id, user_name=user.name) MappedLoans(book_id=2,user_name='Bhargan Basepair',loan_date=None) >>> db.flush() >>> db.loans.delete(db.loans.c.book_id==2) >>> db.loans.update(db.loans.c.book_id==2, book_id=1) >>> db.loans.select_by(db.loans.c.book_id==1) [MappedLoans(book_id=1,user_name='Joe Student',loan_date=datetime....

I don't follow Ruby development too closely (most of the info on it is still in Japanese, after all), but the US RubyConf was held recently so there's been an unusual number of English posts on Ruby, among them David Pollack's The Impending Ruby Fracture. David's article seems to consist of these points: Matz is uninterested in adding static bondage & discipline features to Ruby (true, as far as I know) "Enterprise" users won't be satisfied without said features (more on this below) There are a lot of Ruby runtimes out there right now (the most interesting part of the article) Therefore some Enterprise will co-opt one of the runtimes to fork Ruby and add the B&D features (wtf?) Summarized this way it looks faintly ridiculous, and yet nobody over on the programming reddit has called this out. Maybe I'm taking excessive liberties with David's article, but I don't think I am. The possibility of forking is part of what makes open sou...

I'm pleased that I made it into the round of 1000 in Google's Codejam . The Python support was much appreciated! (Actually, I'd be curious what the breakdown was by language of the contestants and qualifiers. I have no real stats but from the 50+ submissions I glanced at, all C#, Java, and C++, I would guess that topcoder's "Python might be too slow" disclaimer scared a lot of people away from Python. Or, like VB.NET, it just isn't that popular with this group. The ignomy!) I solved the 250 pt problem (problem set 5) very quickly, which is good because I didn't end up solving the 750 pt one at all. The 250 pt problem was You are given a tuple (integer) f that describes a set of values from a function f. The x-th element (zero-indexed) is the value of f(x). The function is not convex at a particular x value if there exist y and z values such that y Here was my brute force solution: class point: def __init__(self, x=0, y=0): self....

If you work in the web development area, or even dabble in it as a hobbyist, sooner or later you're going to write code for a project that needs authentication. Probably sooner than later. For a feature that gets used so frequently, it's remarkable to me that nobody has really done this right. Here are some basic principles for a good solution: A minimum of customization to work out-of-the-box Gentle complexity slope when more sophisticated behavior is needed Play nice with others Don't try to solve world hunger The first two are, I hope, no-brainers. The second two bear more explanation. Play nice with others: not everyone wants to authenticate against a Users table in a relational database. (Fairly common alternatives are LDAP or Unix logins.) If you bake in assumptions like this too deep, it causes problems. It might be worth the problems if it were impossible to provide both generality and ease of use, but such is not the case. Don't try to solve wo...