Spyced

On applying for jobs

2011-11-19T09:24:00.001-08:00

A friend asks,

If [I see a] job I could do, even though I don't meet the stated requirements, should I apply anyway?

Short answer: yes.

Longer answer: companies are all over the map here, although in general the less layers of bureaucracy there are between the team that the candidate will work with and the hiring process, the more likely the list of requirements is to be actual requirements.

How can you tell?

HR paper pushers like to think in terms of checklists because that lets them go through hundreds of resumes without any real understanding of the position, so they write ads like this one -- lots of really specific "5+ years of X," not much about what the position actually involves.

But if it's the team lead himself writing the description, which you will see at smaller companies, then you get much more about what the position involves and less checklist items, because the lead is comfortable determining competence based on skill instead of pattern matching. For a software development position, I don't care if you have a degree in CS if you can code. (Open-source contributions are a better signal for ability and passion than a degree, anyway.) My team has people with no degree, to people with PhDs.

Even when dealing with large companies, you have to factor in that people are terrible at distinguishing "want" from "need." A lot of "requirements" are really "nice-to-haves." It can be tough to tell the difference, but the better idea you have of what the job actually involves, the better you can tell which are hard requirements.

For instance: without knowing anything else about a position, my guess is that "native French speaker" really would be a hard requirement. That's not the sort of thing people tend to put down on a whim. But even then, there are shades of grey. For instance, if I were looking for a job and found a "distributed databases developer position, must know Java, be familiar with open source and be a native French speaker" then I might see if they'd give me a pass on the last part because I'm a really good fit for the rest -- and I know they're unlikely to find a lot of candidates with an exact match.

In short, you have little to lose by trying, but don't just shotgun out resumes; include a cover letter that highlights the best matches from your experience to what they are looking for. Follow up with the hiring manager if possible to ask (a) "I sent in my resume a few days ago, and I wanted to see where you are in the hiring process for this position," and if they reply that they got it but you're not a good fit, ask (b) what specifically they were looking for, so you can flesh out your intuition that much more for next time.

Good luck!

Apache Cassandra: 2010 in review

2011-01-04T11:53:00.000-08:00

In 2010, Apache Cassandra increased its momentum as the leading scalable database. Here is a summary of the notable activity in three areas: code, community and controversy. As always, comments are welcome.

Code

2010 started with the release of Cassandra 0.5, followed by 0.6 and graduation from the ASF incubator a few months later. Seven more stable releases of 0.6 proceeded, adding many features to improve operations in response to feedback from production users.

0.7 adds highly anticipated features like column value indexes, live schema updates, more efficient cluster expansion, and more control over replication, but didn't quite make it into 2010, with rc4 released on new year's 2011.

We also committed the distributed counters patchset, begun at Digg and enhanced by Twitter for their real-time analytics product. Notable as the most-involved feature discussion to date, distributed counters started with a vector clock approach, but switched to a new design by Kelvin Kakugawa after we realized vector clocks were a dead end for anything but the trivial case of monotonic-increments-by-one.

One of the biggest trends was increasing activity around Cassandra as well as in the core database itself. 2010 saw Hadoop map/reduce integration, as well as Pig support and a patch for Hive.

We also saw Lucandra, which implements a Cassandra back end for Lucene and is used in several high volume production sites, grow up into Solandra, embedding Solr and Cassandra in the same JVM for even more performance.

Community

Cassandra hit its stride in 2010, starting with graduation from the ASF incubator in April. 2010 saw 1025 tickets resolved, nearly twice as many compared to 2009 (565).

Like many Apache projects, Cassandra has a relatively small set of committers, but a much larger group of contributors. In 2010 Cassandra passed over 100 people who have contributed at least one patch. Release manager Eric Evans put together a great way to visual this with a Code Swarm video of Cassandra development.

I started Riptano with Matt Pfeil in April to provide professional products and services around Cassandra. In October, we announced funding from Lightspeed and Sequoia. From May to December, we conducted eleven Cassandra training events in eight months, and twice that many private classes on-site with customers.

Riptano is now up to 25 employees, with offices in the San Francisco bay area, Austin, and New York, and engineers working remotely in San Antonio, France, and Belarus.

In August, Riptano and Rackspace organized a very successful inaugural Cassandra Summit, with about 200 attendees (videos available), followed by almost a full track at ApacheCon in November. Cassandra was also represented at many other conferences on multiple subjects, for several languages, and continents.

Controversy

Cassandra got a lot of negative publicity when Kevin Rose blamed Cassandra for Digg v4's teething problems. However, there was no deluge of bug reports coming out of Digg's Cassandra team, and Digg engineers Arin Sarkissian and Chris Goffinet (now working on Cassandra for Twitter) got on Quora to refute the idea that Cassandra was at fault:

The whole "Cassandra to blame" thing is 100% a result of folks clinging on to the NoSQL vs SQL thing. It's a red herring.
The new version of Digg has a whole new architecture with a bunch of technologies involved. Problem is, over the last few months or so the only technological change we mentioned (blogged about etc) was Cassandra. That made it pretty easy for folks to cling on to it as the "problem".

Meanwhile, Digg competitor Reddit has continued migrating to Cassandra, crediting it with enabling their 3x traffic growth in 2010.

More importantly, 2010 saw dozens of new Cassandra deployments, including a new contender for the largest-cluster crown when Digital Reasoning announced a 400-node cluster for the US government.

We look forward to another great year in 2011!

And now for something completely different

2010-04-26T12:29:00.000-07:00

A month ago I left Rackspace to start Riptano, a Cassandra support and services company.

I was in the unusal position of being a technical person looking for a business-savvy co-founder. For whatever reason, the converse seems a lot more common. Maybe technical people tend to sterotype softer skills as being easy.

But despite some examples to the contrary (notably for me, Josh Coates at Mozy), I found that starting a company is too hard for just one person. Unfortunately, all of my fairly slim portfolio of business guys I'd like to co-found with were unavailable. So progress was slow, until Matt Pfeil heard that I was leaving Rackspace and drove to San Antonio from Austin to talk me out of it. Not only was he not successful in talking me out of leaving, but he ended up co-founding Riptano. And here we are, with a Riptano mini-faq.

Isn't Cassandra mostly just a web 2.0 thing for ex-mysql shops?

Although most of the early adopters fit this stereotype, we're seeing interest from a lot of Oracle users and a lot of industries. Unlike many "NoSQL" databases, Cassandra doesn't drop durability (the D in ACID), and besides scalability, enterprises are very interested in our support for multiple data centers and Hadoop analytics.

Are you going to fork Cassandra?

No. Although the ASF license allows doing basically anything with the code, including creating proprietary forks, we think the track record of this strategy in the open source database world is mixed at best.

We might create a (still open-source) Cassandra distribution similar to Cloudera's Distribution for Hadoop, but the mainline Cassandra development is responsive enough that there isn't as much need for a third party to do this as there is with Hadoop.

What does Rackspace think?

Rackspace has been the primary driver of Cassandra development recently, employing (until I left) the three most active committers on the project. For the same reasons Rackspace supported Cassandra to begin with, Rackspace is excited to see Riptano help take the Cassandra ecosystem to the next level. Rackspace has invested in Riptano and has been completely supportive in every way.

Where did you get the name "Riptano?" Does it mean anything?

We took a sophisticated, augmented AI approach. By which I mean, we took a program that generated random, pronouceable strings, and put together a couple fragments that sounded good together. (This is basically the same approach we took at Mozy, only there Josh insisted on a four letter domain name which narrowed it down a lot.)

I hope it doesn't mean "your dog has bad breath" somewhere.

And yes, Riptano is on twitter.

Are you hiring?

Yes. We'll have a jobs page on the site soon. In the meantime you can email me a resume if you can't wait. Prior participation in the Apache Cassandra project is of course a huge plus.

Cassandra: Fact vs fiction

2010-04-07T09:37:00.000-07:00

Cassandra has seen some impressive adoption success over the past months, leading some to conclude that Cassandra is the frontrunner in the highly scalable databases space (a subset of the hot NoSQL category). Among all the attention, some misunderstandings have been propagated, which I'd like to clear up.

Fiction: "Cassandra relies on high-speed fiber between datacenters" and can't reliably replicate between datacenters with more than a few ms of latency between them.

Fact: Cassandra's multi-datacenter replication is one of its earliest features and is by far the most battle-tested in the NoSQL space. Facebook had Cassandra deployed on east and west coast datacenters since before open sourcing it. SimpleGeo's Cassandra cluster spans 3 EC2 availability zones, and Digg is also deployed on both coasts. Claims that this can't possibly work are an excellent sign that you're reading an article by someone who doesn't know what he's talking about.

Fiction: "It’s impossible to tell when [Cassandra] replicas will be up-to-date."

Fact: Cassandra provides consistency when R + W > N (read replica count + write replica count > replication factor), to use the Dynamo vocabulary. If you do writes and reads both with QUORUM, for one example, you can expect data consistency as soon as there are enough reachable nodes for a quorum. Cassandra also provides read repair and anti-entropy, so that even reads at ConsistencyLevel.ONE will be consistent after either of these events.

Fiction: Cassandra has a small community

Fact: Although popularity has never been a good metric for determining correctness, it's true that when using bleeding edge technology, it's good to have company. As I write this late at night (in the USA), there are 175 people in the Cassandra irc channel, 60 in the HBase one, 32 in Riak's, and 15 in Voldemort's. (Six months ago, the numbers were 90, 45, and 12 for Cassandra, HBase, and Voldemort. I did not hang out in #riak yet then.) Mailing list participation tells a similar story.

It's also interesting that the creators of Thrudb and dynomite are both using Cassandra now, indicating that the predicted NoSQL consolidation is beginning.

Fiction: "Cassandra only supports one [keyspace] per install."

Fact: This has not been true for almost a year (June of 2009).

Fiction: Cassandra cannot support Hadoop, or supporting tools such as Pig.

Fact: It has always been straightforward to send the output of Hadoop jobs to Cassandra, and Facebook, Digg, and others have been using Hadoop like this as a Cassandra bulk-loader for over a year. For 0.6, I contributed a Hadoop InputFormat and related code to let Hadoop jobs process data from Cassandra as well, while cooperating with Hadoop to keep processing on the nodes that actually hold the data. Stu Hood then contributed a Pig LoadFunc, also in 0.6.

Fiction: Cassandra achieves its high performance by sacrificing reliability (alternately phrased: Cassandra is only good for data you can afford to lose)

Fact: unlike some NoSQL databases (notably MongoDB and HBase), Cassandra offers full single-server durability. Relying on replication is not sufficient for can't-afford-to-lose-data scenarios; if your data center loses power, you are highly likely to lose data if you are not syncing to disk no matter how many replicas you have, and if you run large systems in production long enough, you will realize that power outages through some combination of equipment failure and human error are not occurrences you can ignore. But with its fsync'd commitlog design, Cassandra can protect you against that scenario too.

What to do after your data is saved, e.g. backups and snapshots, is outside of my scope here but covered in the operations wiki page.

Cassandra in Google Summer of Code 2010

2010-03-30T07:48:00.001-07:00

Cassandra is participating in the Google Summer of Code, which opened for proposal submission today. Cassandra is part of the Apache Software Foundation, which has its own page of guidelines up for students and mentors.

We have a good mix of project ideas involving both core and non-core areas, from straightforward code bashing to some pretty tricky stuff, depending on your appetite. Core tickets aren't necessarily harder than non-core, but they will require reading and understanding more existing code.

Non-core

Create a web ui for cassandra: we have a (fairly minimal) command line interface, but a web gui is more user-friendly. There is the beginnings of such a beast in the Cassandra source tree at contrib/cassandra_browser [pretty ugly Python code] and a gtk-based one at http://github.com/driftx/chiton [also Python, less ugly].
First-class commandline interface: if you prefer to kick things old-school, improving the cli itself would also be welcome.
Create a Cassandra demo application: we have Twissandra, but we can always use more examples to introduce people to "thinking in Casssandra," which is the hardest part of using it. This one seems to be the most popular with students so far. (So stand out from the crowd, and submit something else too. :)

Almost-core

Performance regression tests: pretty self-explanatory?
System tests against multiple nodes: If GSOC were a wish-granting fairy I would probably choose this with my first wish. There's a couple different ways you can approach this; scripting VMs is one, or you could explore the Cassandra simulator that was contributed a while ago (some TLC required).
Hive support: Hive is a project that runs SQL queries against Hadoop map/reduce clusters. (For analytics; it is too high-latency to run applications against Hive directly). ~~HIVE-705~~ added support for backends other than HDFS, with HBase as the first. Cassandra support should be doable too now. The Hive storage backends are described in http://wiki.apache.org/hadoop/Hive/StorageHandlers and the HBase backend specifically in http://wiki.apache.org/hadoop/Hive/HBaseIntegration.

Core

Avro RPC support: currently Cassandra's client layer is the Thrift RPC framework, which sucks for reasons outside our scope here. We're moving to Avro, the new hotness from Doug Cutting (creator of Lucene and Hadoop, you may have heard of those). Basically this means porting org.apache.cassandra.thrift.CassandraServer to org.apache.cassandra.avro.CassandraServer; some examples are already done by Eric Evans.
Session-level consistency: In one and two Amazon discusses the concept of "eventual consistency." Cassandra uses eventual consistency in a design similar to Dynamo. Supporting session consistency would be useful and relatively easy to add: we already have the concept of a Memtable to "stage" updates in before flushing to disk; if we applied mutations to a session-level memtable on the coordinator machine (that is, the machine the client is connected to), and then did a final merge from that table against query results before handing them to the client, we'd get it almost for free.
Optimize commitlog performance: this is about as low-level as you'll find in Cassandra's code base. fsync, CAS, it's all here. http://wiki.apache.org/cassandra/ArchitectureCommitLog describes the current CommitLog design.

You can comment directly on the JIRA tickets after creating an account (it's open to the public) if you're interested or have other questions. And of course feel free to propose other ideas!

Cassandra in action

2010-03-24T08:41:00.001-07:00

There's been a lot of new articles about Cassandra deployments in the past month, enough that I thought it would be useful to summarize in a post.

Ryan King explained in an interview with Alex Popescu why Twitter is moving to Cassandra for tweet storage, and why they selected Cassandra over the alternatives. My experience is that the more someone understands large systems and the problems you can run into with them from an operational standpoint, the more likely they are to choose Cassandra when doing this kind of evaluation. Ryan's list of criteria is worth checking out.

Digg followed up their earlier announcement that they had taken part of their site live on Cassandra with another saying that they've now "reimplemented most of Digg's functionality using Cassandra as our primary datastore." Digg engineer Ian Eure also gave some more details on Digg's cassandra data model in a Hacker News thread.

Om Malik quoted extensively from the Digg announcement and from Rackspace engineer Stu Hood, who explained Cassandra's appeal: "Over the Bigtable clones, Cassandra has huge high-availability advantages, and no single point of failure. When compared to the Dynamo adherents, Cassandra has the advantage of a more advanced datamodel, allowing for a single row to contain billions of column/value pairs: enough to fill a machine. You also get efficient range queries for the top level key, and even within your values."

The Twitter and Digg news kicked off a lot of publicity, including a lot of "me too" articles but some interesting ones, including a highscalability post wondering if this was the end of the mysql + memcached era. If not quite yet the end, then the beginning of it. As Ian Eure from Digg said, "If you're deploying memcache on top of your database, you're inventing your own ad-hoc, difficult to maintain NoSQL system." Possibly the best commentary on this idea is Dare Obasanjo's, who explained "Digg's usage of Cassandra actually serves as a rebuttal to [an article claiming SQL scales just fine] since they couldn't feasibly get what they want with either horizontal or vertical scaling of their relational database-based solution."

Reddit also migrated to Cassandra from memcachedb, in only 10 days, the fastest migration to Cassandra I've seen. More comments from the engineer doing the migration, ketralnis, in the reddit discussion thread.

CloudKick blogged about how they use Cassandra for time series data, including a sketch of their data model. CloudKick migrated from PostgreSQL, skewering the theory you will sometimes see proffered that "only MySQL users are migrating to NoSQL, not people who use [my favorite vendor's relational database]."

Jake Luciani wrote about how Lucandra, the Cassandra Lucene back-end works, and how he's using it to power the Twitter search app sparse.ly. IMO, Lucandra is one of Cassandra's killer apps.

The FightMyMonster team switched from HBase to Cassandra after concluding that "HBase is more suitable for data warehousing, and large scale data processing and analysis... and Cassandra is more suitable for real time transaction processing and the serving of interactive data." Dominic covers CAP, architecture considerations, benchmarks, map/reduce, and durability in explaining his conclusion.

Eric Peters gave a talk on Cassandra use at his company, Frugal Mechanic, at the Seattle Tech Startups Meetup. This was interesting not because Frugal Mechanic is a big name but because it's not. I haven't seen Eric's name on the Cassandra mailing lists at all, but there he was deploying it and giving a talk on it, showing that Cassandra is starting to move beyond early adopters. (And, just maybe, that our documentation is improving. :)

Finally, Eric Florenzano has a live demo up now of Cassandra running a Twitter clone at twissandra.com, with source at github, as an example of how to use Cassandra's data model. If you're interested in the nuts and bolts of how to build an app on Cassandra, you should check it out.

Why your data may not belong in the cloud

2010-03-15T15:19:00.000-07:00

Several of the reports of the recently-concluded NoSQL Live event mentioned that I took a contrarian position on the "NoSQL in the Cloud" panel, arguing that traditional, bare metal servers usually make more sense. Here's why.

There are two reasons to use cloud infrastructure (and by cloud I mean here "commodity VMs such as those provided by Rackspace Cloud Servers or Amazon EC2):

You only need a fraction of the capacity of a single machine
Your demand is highly elastic; you want to be able to quickly spin up many new instances, then drop them when you are done

Most people looking at NoSQL solutions are doing it because their data is larger than a traditional solution can handle, or will be, so (1) is not a very strong motivation. But what about (2)? At first glance, cloud is a great fit for adding capacity to a database cluster painlessly. But there's an important difference between load like web traffic that bounces up and down frequently, and databases: with few exceptions, databases only get larger with time. You won't have 20 TB of data this week, and 2 next.

When capacity only grows in one direction it makes less sense to pay a premium for the flexibility of being able to reduce your capacity nearly instantly, especially when you also get reduced I/O performance (the most common bottleneck for databases) in the bargain because of the virtualization layer. That's why, despite working for a cloud provider, I don't think it's always a good fit for databases. (It doesn't hurt that Rackspace also offers classic bare metal hosting in the same data centers, so you can have the best of both worlds.)

Distributed deletes in the Cassandra database

2010-02-08T10:06:00.000-08:00

Handling deletes in a distributed, eventually consistent system is a little tricky, as demonstrated by the fairly frequent recurrence of the question, "Why doesn't disk usage immediately decrease when I remove data in Cassandra?"

As background, recall that a Cassandra cluster defines a ReplicationFactor that determines how many nodes each key and associated columns are written to. In Cassandra (as in Dynamo), the client controls how many replicas to block for on writes, which includes deletions. In particular, the client may (and typically will) specify a ConsistencyLevel of less than the cluster's ReplicationFactor, that is, the coordinating server node should report the write successful even if some replicas are down or otherwise not responsive to the write.

(Thus, the "eventual" in eventual consistency: if a client reads from a replica that did not get the update with a low enough ConsistencyLevel, it will potentially see old data. Cassandra uses Hinted Handoff, Read Repair, and Anti Entropy to reduce the inconsistency window, as well as offering higher consistency levels such as ConstencyLevel.QUORUM, but it's still something we have to be aware of.)

Thus, a delete operation can't just wipe out all traces of the data being removed immediately: if we did, and a replica did not receive the delete operation, when it becomes available again it will treat the replicas that did receive the delete as having missed a write update, and repair them! So, instead of wiping out data on delete, Cassandra replaces it with a special value called a tombstone. The tombstone can then be propagated to replicas that missed the initial remove request.

There's one more piece to the problem: how do we know when it's safe to remove tombstones? In a fully distributed system, we can't. We could add a coordinator like ZooKeeper, but that would pollute the simplicity of the design, as well as complicating ops -- then you'd essentially have two systems to monitor, instead of one. (This is not to say ZK is bad software -- I believe it is best in class at what it does -- only that it solves a problem that we do not wish to add to our system.)

So, Cassandra does what distributed systems designers frequently do when confronted with a problem we don't know how to solve: define some additional constraints that turn it into one that we do. Here, we defined a constant, GCGraceSeconds, and had each node track tombstone age locally. Once it has aged past the constant, it can be GC'd. This means that if you have a node down for longer than GCGraceSeconds, you should treat it as a failed node and replace it as described in Cassandra Operations. The default setting is very conservative, at 10 days; you can reduce that once you have Anti Entropy configured to your satisfaction. And of course if you are only running a single Cassandra node, you can reduce it to zero, and tombstones will be GC'd at the first compaction.

Cassandra 0.5.0 released

2010-01-25T14:23:00.000-08:00

Apache Cassandra 0.5.0 was released over the weekend, four months after 0.4. (Upgrade notes; full changelog.) We're excited about releasing 0.5 because it makes life even better for people using Cassandra as their primary data source -- as opposed to a replica, possibly denormalized, of data that exists somewhere else.

The Cassandra distributed database has always had a commitlog to provide durable writes, and in 0.4 we added an option to waiting for commitlog sync before acknowledging writes, for cases where even a few seconds of potential data loss was not an option. But what if a node goes down temporarily? 0.5 adds proactive repair, what Dynamo calls "anti-entropy," to synchronize any updates Hinted Handoff or read repair didn't catch across all replicas for a given piece of data.

0.5 also adds load balancing and significantly improves bootstrap (adding nodes to a running cluster). We've also been busy adding documentation on operations in production and system internals.

Finally, in 0.5 we've improved concurrency across the board, improving insert speed by over 50% on the stress.py benchmark (from contrib/) on a relatively modest 4-core system with 2GB of ram. We've also added a [row] key cache, enabling similar relative improvements in reads:

(You will note that unlike most systems, Cassandra reads are usually slower than writes. 0.6 will narrow this gap with full row caching and mmap'd I/O, but fundamentally we think optimizing for writes is the right thing to do since writes have always been harder to scale.)

Log replay, flush, compaction, and range queries are also faster.

0.5 also brings new tools, including JSON-based data export and import, an improved command-line interface, and new JMX metrics.

One final note: like all distributed systems, Cassandra is designed to maximize throughput when under load from many clients. Benchmarking with a single thread or a small handful will not give you numbers representative of production (unless you only ever have four or five users at a time in production, I suppose). Please don't ask "why is Cassandra so slow" and offer up a single-threaded benchmark as evidence; that makes me sad inside. Here's 1000 words:

(Thanks to Brandon Williams for the graphs.)

Linux performance basics

2010-01-21T08:22:00.000-08:00

I want to write about Cassandra performance tuning, but first I need to cover some basics: how to use vmstat, iostat, and top to understand what part of your system is the bottleneck -- not just for Cassandra but for any system.

vmstat

You will typically run vmstat with "vmstat sampling-period", e.g., "vmstat 5." The output looks like this:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
20  0 195540  32772   6952 576752    0    0    11    12   38   43  1  0 99  0
22  2 195536  35988   6680 575132    6    0  2952    14  959 16375 72 21  4  3

The first line is your total system average since boot; typically this will not be very useful, since you are interested in what is causing problems NOW. Then you will get one line per sample period; most of the output is self explanatory. The reason to start with vmstat is the "swap" section: si and so are swap in (memory read from disk) and swap out (memory written to disk). Remember that a little swapping is normal, particularly during application startup: by default, Linux will swap infrequently used pages of application memory to disk to free up more room for disk caching, even if there is enough ram to accommodate all applications.

iostat

To get more details of io, use iostat -x. Again, you want to give it a sampling interval, and ignore the first set of output. iostat also gives you some cpu information but top does that better; let's focus on the Device section:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               9.80     0.20   36.60    0.40  5326.40     4.80   144.09     0.06    1.62   1.41   5.20

There are 3 easy ways to tell if a disk is a probable bottleneck here, and none of them show up without the -x flag, so get in the habit of using that. "avgqu-sz" is the size of the io request queue; if it is large, there are lots of requests waiting in line. "await" is how long (in ms) the average request took to be satisfied (including time enqueued); recall that on non-SSDs, a single seek is between 5 and 10ms. Finally, "%util" is Linux's guess at how fully saturated the device is.

top

To learn more about per-process CPU and memory usage, use "top." I won't paste top output here because everyone is so familiar with it, but I will mention a few useful things to know:

"P" and "M" toggle between sorting by cpu usage and sorting by memory usage
"1" toggles breaking down the CPU summary by CPU core
SHR (shared memory) is included in RES (resident memory)
Amount of memory belonging to a process that has been swapped out is VIRT - RES
a state (S column) of D means the process (or thread, see below) is waiting for disk or network i/o
"steal" is how much CPU the hypervisor is giving to another VM in a virtual environment; as virtual provisioning becomes more common, avoiding noisy neighbors is increasingly important

"top -H" will split out individual threads into their own lines; both per-process and per-thread views are useful. The per-thread view is particularly useful when dealing with Java applications since you can easily correlate them with thread names from the JVM to see which threads are consuming your CPU. Briefly, you take the PID (thread ID) from top, convert it to hex -- e.g., "python -c 'print hex(12345)'" -- and match it with the corresponding thread ID from jstack.

Now you can troubleshoot with a process like: "Am I swapping? If so, what processes are using all the memory? If my application makes a lot of disk read requests, are my reads being cached or are they actually hitting the disk? If I am hitting the disk, is it saturated? How much 'hot data' can I have before I run out of cache room? Are any/all of my cpu cores maxed? Which threads are actually using the CPU? Which threads spend most of their time waiting for i/o?" Then if you go to ask for help tuning something, you can show that you've done your homework.

Cassandra reading list

2009-12-15T14:01:00.001-08:00

I put together this list for a co-worker who wants to learn more about Cassandra: (0.5 beta 2 out now!)

Getting Started: Cassandra is surprisingly easy to try out. This walks you through both single-node and clustered setup.
The Dynamo paper and Amazon's related article on eventual consistency: Cassandra's replication model is strongly influenced by Dynamo's. Almost everything you read here also applies to Cassandra. (The major exceptions are vector clocks, and even that may change, and Cassandra's support for order-preserving partitioning with active load balancing.)
WTF is a SuperColumn? Arin Sarkissian from Digg explains the Cassandra data model.
Operations: stuff you will want to know when you run Cassandra in production
Cassandra users survey from Nov 09: What Twitter, Mahalo, Ooyala, SimpleGeo, and others are using Cassandra for
More articles here (Cassandra on OS X seems to be a particularly popular topic)

If you want to know more about the internals, also see these:

Internals documentation
Facebook presentation and NoSQL SF presentation, by Avinash Lakshman (the second picks up almost where the first leaves off)
LADIS 2009 paper by Avinash Lakshman and Prashant Malik

Cassandra hackfest and OSCON report

2009-07-29T10:34:00.000-07:00

The best part of OSCON for me wasn't actually part of OSCON. The guys at Twitter put together a Cassandra hackfest on Wednesday night, with much awesomeness resulting. Thanks to Evan for organizing!

Stu Hood flew up from Rackspace's Virginia offices just for the night, which normally probably wouldn't have been worth it, but Cliff Moon, author of dynomite, showed up (thanks, Cliff!) and was able to give Stu a lot of pointers on implementing merkle trees. Cliff and I also had a good discussion with Jun Rao about hinted handoff--Cliff and Jun are not fans, and I tend to agree with them--and eventual consistency.

I also met David Pollack and got to talk a little about persistence for Goat Rodeo, and talked to a ton of people from Twitter and Digg. I think those two, with Rackspace and IBM Research, constituted the companies with more than one engineer attending. The rest was "long tail."

Back at OSCON, my Cassandra talk was standing room only. Slides:

Cassandra: Open Source Bigtable + Dynamo

View more documents from jbellis.

My second talk is the one I would have preferred to give first, on "What Every Developer Should Know About Database Scalability". (I would have preferred to give it first so that I could have just said "come to my Cassandra talk for more details" instead of trying to cram that in at the end. But, it was in my proposal outline!) Slides:

What Every Developer Should Know About Database Scalability

View more documents from jbellis.

Other OSCON talks I liked (that have slides available):

Gearman: Bringing the Power of Map/Reduce to Everyday Applications
High Performance SQL with PostgreSQL [8.4]
Linux Filesystem Performance for Databases (reiserfs blows everyone away for random writes, by a factor of > 2!?)
Neo4j - The Benefits of Graph Databases
Release Mismanagement: How to Alienate Users and Frustrate Developers

Cassandra 0.3 update

2009-07-06T10:56:00.000-07:00

Two months after the first release candidate, Cassandra 0.3 is still not out. But, we're close!

We had two more bug-fix release candidates, and it's virtually certain that 0.3-final will be the same exact code as 0.3-rc3. (If you're using rc1, you do want to upgrade; see CHANGES.txt.) But, we got stuck in the ASF bureaucracy and it's going to take at least one more round-trip before the crack Release Prevention Team grudgingly lets us call it official.

In the meantime, work continues apace on trunk for 0.4.

Patch-oriented development made sane with git-svn

2009-06-23T09:40:00.000-07:00

One of the drawbacks to working on Cassandra is that unlike every other OSS project I have ever worked on, we are using a patch-oriented development process rather than post-commit review. It's really quite painfully slow. Somehow this became sort of the default for ASF projects, but there is precedent for switching to post-commit review eventually.

In the meantime, there is git-svn.

(The ASF does have a git mirror set up, but I'm going to ignore that because (a) its reliability has been questionable and (b) sticking with git-svn hopefully makes this more useful for non-ASF projects.)

Disclaimer: I am not a git expert, and probably some of this will make you cringe if you are. Still, I hope it will be useful for some others fumbling their way towards enlightenment. As background, I suggest the git crash course for svn users. Just the parts up to the Remote section.

Checkout:

git-svn init https://svn.apache.org/repos/asf/cassandra/trunk cassandra

Once that's done the only git-svn commands you need to know about are dcommit to push the changes in the current git branch back to svn, and rebase, to pull changes from svn and re-apply your uncommitted patches on top of that (basically exactly like svn up).

Creating new code:

git checkout -b [ticket number]
[edit stuff, maybe get add or git rm new or obsolete files]
git commit -a -m 'commit'
repeat 2-3 as necessary
git-jira-attacher [revision] (usually some variant of HEAD^^^)

[after review]

git log (just to make sure I'm about to commit what I think I'm about to commit)
git-svn dcommit
git checkout master
git-svn rebase -l (this will put the changes you just committed into master)
git branch -d [ticket number]

When I'm reviewing code it looks similar:

git checkout -b [ticket number]
wget patches and git-apply, or jira-apply CASSANDRA-[ticket-number]
review in gitk/qgit and/or IDE (the intellij git plugin is quite decent)
commit .. branch -d as above

The last operation is "see who I need to bug to get reviews moving." This is just a list of the branches I haven't merged into master and deleted yet:

git branch

Git-svn takes a lot of the pain out of the ASF's patch-and-jira workflow. In particular, you can easily break changes for a ticket up into multiple patches that are easily reviewed, and the latency of waiting for patch review doesn't kill your throughput so badly since you can just leave that branch alone and start a new one for your next piece of functionality. And of course you get git commit --amend and git rebase -i for massaging patches during the review process.

One fairly common complication is if you finish a ticket A, then start on ticket B (that depends on A) while waiting for A to be reviewed. So you checkout -b from your branch A rather than master and build some patches on that. As sometimes happens, the reviewer finds something you need to improve in your patch set for A, so you make those changes. Now you need to rebase your patches to B on top of the changes you made to A. The best way to do this is to branch A to B-2, then git cherry-pick from B and resolve conflicts as necessary.

Final note: I often like to create lots of small commits as I am exploring a solution and combine them into larger units with git rebase -i for patch submission. (It's easier to combine small patches, than pull apart large ones.) So my early commit messages are often terse and need editing. You can change commit messages with edit mode in rebase, then using commit --amend and rebase --continue, but that is tedious. I complained about this to my friend Zach Wily and he made this git amend-message command (place in [alias] in your .gitconfig):

   amend-message = "!bash -c ' \
       c=$0; \
       if [ $c == \"bash\" ]; then echo \"Usage: git amend-message <commit>\"; exit 1; fi; \
       saved_head=$(git rev-parse HEAD); \
       commit=$(git rev-parse $c); \
       commits=$(git log --reverse --pretty=format:%H $commit..HEAD); \
       echo \"Rewinding to $commit...\"; \
       git reset --hard $commit; \
       git commit --amend; \
       for X in $commits; do \
           echo \"Applying $X...\"; \
           git cherry-pick $X >> /dev/null; \
           if [ $? -ne 0 ]; then \
               echo \"  apply failed (is this a merge?), rolling back all changes\"; \
               git reset --hard $saved_head; \
               echo \" ** AMEND-MESSAGE FAILED, sorry\"; \
               exit 1; \
           fi; \
       done; \
       echo \"Done\"'"

(Zach would like the record to show that he knows this is pretty hacky. "For instance, it won't work if one of the commits after the one you're changing is a merge, since cherry-pick can't handle those." But it's quite useful, all the same.)

For what it's worth, the rest of my aliases are

 st = status
 ci = commit
 co = checkout
 br = branch
 cp = cherry-pick

Why you won't be building your killer app on a distributed hash table

2009-05-27T06:59:00.000-07:00

I ran across A case study in building layered DHT applications while doing some research on implementing load-balancing in Cassandra. The question addressed is, "Are DHTs a general-purpose tool that you can build more sophisticated services on?"

Short version: no. A few specialized applications can and have been built on a plain DHT, but most applications built on DHTs have ended up having to customize the DHT's internals to achieve their functional or performance goals.

This paper describes the results of attempting to build a relatively complex datastructure (prefix hash trees, for range queries) on top of OpenDHT. The result was mostly failure:

A simple put-get interface was not quite enough. In particular, OpenDHT relies on timeouts to invalidate entries and has no support for atomicity primitives... In return for ease of implementation and deployment, we sacrificed performance. With the OpenDHT implementation, a PHT query operation took a median of 2–4 seconds. This is due to the fact that layering entirely on top of a DHT service inherently implies that applications must perform a sequence of put-get operations to implement higher level semantics with limited opportunity for optimization within the DHT.

In other words, there are two primary problems with the DHT approach:

Most DHTs will require a second locking layer to achieve correctness when implementing a more complex data structure on top of the DHT semantics. In particular, this will certainly apply to eventually-consistent systems in the Dynamo mold.
Advanced functionality like range queries needs to be supported natively to be at all efficient.

While they spin this in a positive manner -- "hey, at least it didn't take much code" -- the reality is that for most of us, query latency of two to four seconds is several orders of magnitude away from acceptable.

This is one reason why I think Cassandra is the most promising of the open-source distributed databases -- you get a relatively rich data model and a distribution model that supports efficient range queries. These are not things that can be grafted on top of a simpler DHT foundation, so Cassandra will be useful for a wider variety of applications.

Belated 2009 Introduction to SQLAlchemy slides

2009-05-18T15:19:00.001-07:00

I was asked to put my slides up again -- sorry it took so long. The slides and code samples are now up here. Video of the tutorial is also up. (3 parts, first is linked). There's definitely audio problems in parts but at least some is watchable.

Cassandra 0.3 release candidate and progress

2009-05-13T20:18:00.000-07:00

We have a release candidate out for Cassandra 0.3. Grab the download and check out how to get started. The facebook presentation from almost a year ago now is also still a good intro to some of the features and data model.

Cassandra in a nutshell:

Scales writes very, very well: just add more nodes!
Has a much richer data model than vanilla key/value stores -- closer to what you'd be used to in a relational db.
Is pretty bleeding edge -- to my knowledge, Facebook is the only group running Cassandra in production. (Their largest cluster is 120 machines and 40TB of data.) At Rackspace we are working on a Cassandra-based app now that 0.3 has the extra features we need.
Moved to the Apache Incubator about 40 days ago, at which point development greatly accelerated.

Changes in 0.3 include

Range queries on keys, including user-defined key collation.
Remove support, which is nontrivial in an eventually consistent world.
Workaround for a weird bug in JDK select/register that seems particularly common on VM environments. Cassandra should deploy fine on EC2 now. (Oddly, it never had problems on Slicehost / Cloud Servers, which is also Xen-based.)
Much improved infrastructure: the beginnings of a decent test suite ("ant test" for unit tests; "nosetests" for system tests), code coverage reporting, etc.
Expanded node status reporting via JMX
Improved error reporting/logging on both server and client
Reduced memory footprint in default configuration
and plenty of bug fixes.

For those of you just joining us, Cassandra already had

An advanced on-disk storage engine that never does random writes
Transaction log-based data integrity
P2P gossip failure detection
Read repair
Hinted handoff
Bootstrap (adding new nodes to a running cluster)

(Read repair and hinted handoff are discussed in more detail in the Dynamo paper.)

The cassandra development and user community is also growing at an exciting pace. Besides the original two developers from Facebook, we now have five developers regularly contributing improvements and fixes, and many others on a more ad-hoc basis.

How fast is it?

In a nutshell, Cassandra is much faster than relational databases, and much slower than memory-only systems or systems that don't sync each update to disk. Actual benchmarks are in the works. We plan to start performance tuning with the next release, but if you want to benchmark it, here are some suggestions to get numbers closer to what you'll see in the wild (and about 10x more throughput than if you don't do these):

Do enough runs of your benchmark first that each operation tested by your suite runs 20k times before timing it for real. This will allow the JVM jit to compile down to machine code; otherwise you'll just be getting the interpreted version.
Change the root logger level in conf/log4j.properties from DEBUG to INFO; we do a LOT of logging for debuggability and for small column values the logging has more overhead than the actual workload. (It would be even faster if we were to remove them entirely but that didn't make this release.)

A better analysis of Cassandra than most

2009-05-01T13:54:00.000-07:00

Vladimir Sedach wrote a three-part dive into Cassandra. (Almost two months ago now. Guess I need to set up Google Alerts. Trouble is there's a surprising amount of noise around the word `cassandra.`)

A few notes:

We now have an order-preserving partitioner as well as the hash-based one
Yes, if you tell Cassandra to wait for all replicas to be ack'd before calling a write a success, then you would have traditional consistency (as opposed to "eventual") but you'd also have no tolerance for hardware failures which is a main point of this kind of system.
Zookeeper is not currently used by Cassandra, although we have plans to use it in the future.
Load balancing is not implemented yet.
The move to Apache is finished and development is active there now.

Consistent hashing vs order-preserving partitioning in distributed databases

2009-05-01T07:44:00.000-07:00

The Cassandra distributed database supports two partitioning schemes now: the traditional consistent hashing scheme, and an order-preserving partitioner.

The reason that almost all similar systems use consistent hashing (the dynamo paper has the best description; see sections 4.1-4.3) is that it provides a kind of brain-dead load balancing from the hash algorithm spreading keys across the ring. But the dynamo authors go into some detail about how this by itself doesn't actually give good results in practice; their solution was to assign multiple tokens to each node in the cluster and they describe several approaches to that. But Cassandra's original designer considers this a hack and prefers real load balancing.

An order-preserving partitioner, where keys are distributed to nodes in their natural order, has huge advantages over consistent hashing, particularly the ability to do range queries across the keys in the system (which has also been committed to Cassandra now). This is important, because the corollary of "the partitioner uses the key to determine what node the data is on" is, "each key should only have an amount of data associated with it (see the data model explanation) that is relatively small compared to a node's capacity." Cassandra column families will often have more columns in them than you'd see in a traditional database, but "millions" is pushing it (depending on column size) and "billions" is a bad idea. So you'll want to model things such that you spread data across multiple keys and if you then pick an appropriate key naming convention, range queries will let you slice and dice that data as needed.

Cassandra is in the process of implementing load balancing still, but in the meantime order-preserving partitioning is still be useful without that if you know what your key distribution will look like in advance and can pick your node tokens accordingly. Otherwise, there's always the old-school hash-based partitioner until we get that done (for the release after the one we'll have in the next week or so).

See the introduction and getting started pages of the Cassandra wiki for more on Cassandra, and drop us a line on the mailing list or in IRC if you have questions; we're actively trying to improve our docs.

Automatic project structure inference

2009-04-30T14:58:00.000-07:00

David MacIver has an interesting blog entry up about determining logical project structure via commit logs. I was very interested because one of Cassandra's oldest issues is creating categories for our JIRA instance. (I've never been a big fan of JIRA, but you work with the tools you have. Or the ones the ASF inflicts on you, in this case.)

The desire to add extra work to issue reporting for a young project like Cassandra strikes me as slightly misguided in the first place. I have what may be an excessive aversion to overengineering, and I like to see a very clear benefit before adding complexity to anything, even an issue tracker. Still, I was curious to see what David's clustering algorithm made of things. And after pestering him to show me how to run his code I figure I owe it to him to show my results.

In general it did a pretty good job, particularly with the mid-sized groups of files. The large groups are just noise; the small groups, well, it's not exactly a revelation that Filter and FilterTest go together. I'd be tempted to play with it more but with only about two months and 250 commits in the apache repo there's not really all that much data there. (Cassandra's first two years were in an internal Facebook repository.) Working with data that exists as a side effect of natural activity is fascinating.

The best PyCon talk you didn't see

2009-04-11T06:52:00.000-07:00

There were a lot of good talks at PyCon but I humbly submit that the best one you haven't seen yet is Robert Brewer's talk on DejaVu. Robert describes how his Geniusql layer disassembles and parses python bytecode to let his ORM turn python lambdas into SQL. Microsoft got a lot of press for doing something similar for .NET with LINQ, but Bob was there first.

  box = store.new_sandbox()
print [c.Title for c in box.recall(
  Comic, lambda c: 'Hob' in c.Title or c.Views > 0)]

This is cool as hell. The Geniusql part start about 15 minutes in.

Credit where credit is due

2009-04-07T07:57:00.000-07:00

I'm starting to conclude that git just doesn't fit my brain. Several months in, I'm still confused when things don't work the way they "should." My co-worker says I should start a wiki for weird-ass things to do with git: "You keep coming up with use cases that would never occur to me." But, I have to give the git community credit: I've never gone in to #git on freenode and gotten less than fantastic help. Even with git-svn.

Distributed Databases and Cassandra at PyCon

2009-03-28T08:29:00.001-07:00

I'll be leading an open-spaces discussion about distributed database architecture, implementation, and use today at 5:00 PM in the Lambert room. Specifically, we will cover bigtable, dynamo, and cassandra, and how to port a typical relational schema to cassandra's ColumnFamily model.

I wrote a little background information yesterday about why I think Cassandra in particular is compelling.

Why I like the Cassandra distributed database

2009-03-27T06:17:00.000-07:00

I need a distributed database. A real distributed database; replication doesn't count because under a replication-oriented db, each node still needs to be able to handle the full write volume, and you can only throw hardware at that for so long.

So, I'm working on the Cassandra distributed database. I gave a lightning talk on it at PyCon this morning. Cassandra is written in Java and implements a sort of hybrid between Dynamo and Bigtable. (Both papers are worth reading.) It takes its distribution algorithm from Dynamo and its data model from Bigtable -- sort of the best of both worlds. Avinash Lakshman, Cassandra's architect, is one of the authors of the Dynamo paper.

There is a video about Cassandra here. The first 1/4 is about using Cassandra and then the rest is mostly about the internals.

Cassandra is very bleeding edge. Facebook runs several Cassandra clusters in production (the largest is 120 machines and 40TB of data), but there are sharp edges that will cut you. If you want something that Just Works out of the box Cassandra is a poor fit right now and will be for several months.

Cassandra was open-sourced by Facebook last summer. There was some initial buzz but the facebook developers had trouble dealing with the community and the project looked moribund -- FB was doing development against an internal repository and throwing code over the wall every few months which is no way to run an OSS project. Now the code is being developed in the open on Apache and I was voted in as a committer so things are starting to move again.

There are other distributed databases that are worth considering. Here is why those don't fit my needs (and I am not saying that these are bad choices if your requirements are different, just that they don't work for me):

HBase:

Follows the bigtable model, so it's more complicated than it needs to be. (300+kloc vs 50 for Cassandra; many more components). This means it's that much harder for me to troubleshoot. HBase is more bug-free than Cassandra but not so bug-free that troubleshooting would not be required.
Does not have any non-java clients. I need CPython support.
Sits on top of HDFS, which is optimized for streaming reads, not random accesses. So HBase is fine for batch processing but not so good for online apps.

Hypertable:

See HBase, except it's written in C++.

Project Voldemort:

Voldemort is probably the key / value store in the pattern of Dynamo that is farthest along. If you only need key / value I would definitely recommend Voldemort. Next closest is probably Dynomite. Then there are a whole bunch of "me too" key value stores that made fatal architecture decisions or are writing a "memcached with persistence" without really thinking it through.

Running Cassandra [updated 04-22-09 for hacker news visitors]:

prereqs: jdk6 and ant
Check out the code from http://svn.apache.org/repos/asf/incubator/cassandra/trunk (did I mention this was early-adopter only?)
run ant [optional: ant test]. If you get an error like "class file has wrong version 50.0, should be 49.0" then ant is using an old jdk version instead of 6.
For non-java clients, install Thrift. (For java, trunk includes libthrift.jar.) This is a major undertaking in its own right. See this page for a list of dependencies, although most of the rest of that page is now outdated -- for instance, Cassandra no longer depends on the fb303 interface. Python users will have to hand-edit the generated Cassandra.py in three obvious places until this bug is fixed -- just replace the broken argument with None.
run bin/cassandra [optionally -f for foreground]
Connect to the server. By default it listens on localhost:9160. Look at config/server-conf.xml for the columnfamily definitions.
Insert and query some data. Here is an introduction.
Ask on the mailing list or #cassandra on freenode if you have questions.

(Cassandra has a new website up to replace the google code one. We're actively working on the docs, so let us know what needs work.) Good luck!

Impressed by KDE 4.2

2009-02-16T07:58:00.000-08:00

I'm running Linux on my desktop at work after a year of OS X, and Gnome as shipped by Ubuntu 8.10 has just been a world of hurt. The panel looks and works like ass when moved to the left side of the screen (the only sane place to put it in today's world of widescreen monitors), network-manager just decided to quit working one day (I got by with wicd after that), alt-tab behavior sucks both ways you can configure it, etc.

I installed KDE 4.2 over the weekend to see if I was missing anything there.

Wow.

It's like daylight after being in a cave for two months. I didn't realize how hard it has been to use a butt-ugly environment until I wasn't anymore. (Yes, I tried all the gnome themes I could find. Even Nimbus which took a bit of work. What's that recently-famous phrase? "Lipstick on a pig?")

What is better in KDE? In a word, everything. And put me in the camp that really likes having the desktop turned into a usable area for the first time. Like apple's dashboard, except it doesn't suck. I always hated dashboard.

Things that could be improved:

Never in a thousand years would I have thought to look under "Regional & Language" for the preference to turn caps lock into control. I had to google this.
I'm still not sure how to set F9 to Present Windows. Or how to bind a keystroke to the K menu as a poor man's quicksilver.
More generally, a "Welcome to kde. Let me teach you how to be a power user" tutorial would be nice. I have the feeling there is lots of awesome under the hood if I knew where it was. I never got that feeling from gnome. ("Beauty is only skin deep, but ugly goes right to the bone.")
Firefox UI widgets are imperfectly themed from XUL to GTK to KDE. But it is useable. (And having my second monitor redraw correctly instead of leaving artifacts when windows are moved makes up for that.) Is this KDE's fault? Firefox's? I don't know.
Konqueror is still using KHTML instead of webkit which means it is mostly unusable in the world of "web 2.0." Yes, you can install webkitkde but that is Very Alpha. ("Open in new window" doesn't work, for instance. "Open in new tab" is gone entirely.)
I couldn't find an option to just use icons in the task manager widget.

SQLAlchemy at PyCon 2009

2009-02-07T08:47:00.000-08:00

I will be giving an Introduction to SQLAlchemy tutorial and Mike Bayer and Jason Kirtland will be teaching Advanced SQLAlchemy, both on Thursday. I'll be covering similar material as last year, updated for 0.5. I'm also trying to see if I can get the emails of the registrants so far to see what else they would like covered. My tutorial style is exercise-heavy, so if you've read the docs or my slides but still find it hard to write SQLA code, coming to the tutorial is a great way to fix that. (Note: the blog link to the 2008 slides is broken since we moved utahpython.org. If you want them, drop me a note.)

All you ever wanted to know about writing bloom filters

2009-01-28T15:17:00.000-08:00

It seems like Bloom filters are all the rage these days. Three years ago I had barely heard of them and now it seems like I see articles and code using them all the time. That's mostly a good thing, since bloom filters are a very useful tool to avoid performing expensive computations without the full memory overhead of a standard map/dictionary.

Bloom filters are surprisingly simple: divide a memory area into buckets (one bit per bucket for a standard bloom filter; more -- typically four -- for a counting bloom filter). To insert a key, generate several hashes per key, and mark the buckets for each hash. To check if a key is present, check each bucket; if any bucket is empty, the key was never inserted in the filter. If all buckets are non-empty, though, the key is only probably inserted -- other keys' hashes could have covered the same buckets. Determining exactly how big to make the filter and how many hashes to use to achieve a given false positive rate is a solved problem; the math is out there.

But it turns out that it's surprisingly hard to find good information on one part of the implementation: how do you generate an indefinite number of hashes? Even small filters will use three or four; a dozen or more is not unheard of.

Cassandra uses bloom filters to save IO when performing a key lookup: each SSTable has a bloom filter associated with it that Cassandra checks before doing any disk seeks, making queries for keys that don't exist almost free. Unfortunately, Cassandra was using perhaps the worst possible implementation of hash functions: a hardcoded list of hash functions, apparently from this page. A lot of those hashes are reimplementations of the same algorithm! For 6 or less hash functions, Cassandra was only actually generating 2 distinct hash values, so the false positive rate was far far higher than would have been expected.

Judging from the bloom filter implementations out there, generating appropriate hashes is surprisingly hard to get right. One implementation in java uses object.hashCode() to seed the stdlib's PRNG and calls Random.nextInt() to generate the bloom hashes. This works okay for small filters but the false positive rate is up to 140% of the expected rate for large filters. This one in python combines the stdlib hash and a pjw hash with simple arithmetic to achieve "only" an extra 10-15% false positives.

It turns out that most off-the-shelf algorithms suck for bloom filter purposes because their output isn't distributed well-enough across the (32-bit) hash space; most hash function authors are more concerned with speed than achieving a really good output distribution. For a lot of applications, this is a good tradeoff, but you are presumably using a bloom filter because a filter hit requires a relatively very expensive operation, so double-digit increases in this rate is Very Bad. (So I would be skeptical of any home-made hash function generator too, like PyBloom's.)

There are two approaches that I found generate results in line with the theoretical predictions. One is to use a cryptographic hash function like SHA-1. Cryptography's requirements may be overkill for a bloom filter, but crypto hashes do spread their output uniformly across the hash space. So SHA-1 works fine, and you can feed the hash a constant or part of the earlier-generated hash to generate extra data as needed so you're not limited to 5 hashes. (Although Java makes this a pain by having all the digest methods also reset the hash to uninitialized. Explicit is better than implicit!)

The other is the jenkins and murmur hashes, which are used by hadoop's bloom filter implementation. You get as many hashes as you want by using the output of hash#i as the initial value for hash#i+1. Both of these generate very well-distributed hashes, and are about twice as fast as the SHA approach, respectively, for 5 hashes, with murmur being about 10% faster than jenkins. (Remember, SHA will give you 160 bits of pseudo-randomness at a time, so these hashes will be faster still depending on how far you are from a multiple of that.)

In short: use the murmur hash in your bloom filter. If you're using java, you can grab Cassandra's implementation from my github tree until our new repository is up at the apache incubation site. Unfortunately it's spread across half a dozen files, without even counting the tests. Start with [Counting]BloomFilter and go up from there. Your other option is Hadoop's (if you grab it from svn, because that has my fix included), which is even less self-contained and requires wrapping your keys in a Key object. Sorry.

Bonus tip: if you do roll your own, you should probably port Cassandra's test suite to your language of choice. In particular, nobody else bothers to check that false positive rates are within what is predicted by the math, but if you don't, it's easy for bugs that pass simple tests to slip by, like the one in Cassandra's older implementation, or the one I found in Hadoop's.

CouchDB: not drinking the kool-aid

2008-12-31T08:16:00.000-08:00

This is my attempt to clear up some misconceptions about CouchDB and point out some technical details that a lot of people seem to have overlooked. For the record, I like Damien Katz's blog, he seems like a great programmer, and Erlang looks cool. Please don't hurt me.

First, and most important: CouchDB is not a distributed database. BigTable is a distributed database. Cassandra and dynomite are distributed databases. (And open source, and based on a better design than BigTable. More on this in another post.) It's true that with CouchDB you can "shard" data out to different instances just like you can with MySQL or PostgreSQL. That's not what people think when they see "distributed database." It's also true that CouchDB has good replication, but even multi-master replication isn't the same as a distributed database: you're still limited to the write throughput of the slowest machine.

Here are some reasons you should think twice and do careful testing before using CouchDB in a non-toy project:

Writes are serialized. Not serialized as in the isolation level, serialized as in there can only be one write active at a time. Want to spread writes across multiple disks? Sorry.
CouchDB uses a MVCC model, which means that updates and deletes need to be compacted for the space to be made available to new writes. Just like PostgreSQL, only without the man-years of effort to make vacuum hurt less.
CouchDB is simple. Gloriously simple. Why is that a negative? It's competing with systems (in the popular imagination, if not in its author's mind) that have been maturing for years. The reason PostgreSQL et al have those features is because people want them. And if you don't, you should at least ask a DBA with a few years of non-MySQL experience what you'll be missing. The majority of CouchDB fans don't appear to really understand what a good relational database gives them, just as a lot of PHP programmers don't get what the big deal is with namespaces.
A special case of simplicity deserves mention: nontrivial queries must be created as a view with mapreduce. MapReduce is a great approach to trivially parallelizing certain classes of problem. The problem is, it's tedious and error-prone to write raw MapReduce code. This is why Google and Yahoo have both created high-level languages on top of it (Sawzall and Pig, respectively). Poor SQL; even with DSLs being the new hotness, people forget that SQL is one of the original domain-specific languages. It's a little verbose, and you might be bored with it, but it's much better than writing low-level mapreduce code.

RackLabs

2008-12-24T07:42:00.000-08:00

Today marks one month that I've been working for Rackspace's RackLabs with the Mosso group in San Antonio, Texas. (Anyone want to start a Python group? The closest one is in Austin.)

It's kind of a gentle introduction to big company culture for me; at around 2,000 employees, Rackspace is easily ten times as large as any other company I've worked for, and 100 times as large as most. Mosso is a lot smaller and RackLabs itself is smaller still, but I still had to go to five days (!) of corporate orientation. Other than that, though, we're pretty much left alone by our corporate parent.

To start with, I'm working on Mosso's Cloud Files, which is basically an S3 competitor. Cloud Files is similar to the work I did at Mozy, but there are a lot of technical differences. Some are driven by Cloud Files being more of a general purpose storage engine than the one I wrote for Mozy; others stem from the Cloud Files authors being Twisted fans.

Strange coincidence: as with Mozy, I share an office here with a Debian developer, probably the only one in San Antonio. My experience is that debian developers are pretty sharp guys, probably in no small part due to the rigorous screening process you have to go through. They set a high bar.

Of course this continues to be my personal blog, and all opinions are mine alone. RackLabs has its own blog for when they want to say something official.

Frustrated with git

2008-12-19T21:47:00.000-08:00

I'm a little over a week into a git immersion program. Let me just say that git's reputation of being a little arcane (okay, more than a little) and having a steep learning curve is 100% deserved.

One thing that would mitigate things is if git would give you feedback when you tell it to do nonsense. But it doesn't. Here's me trying to get machine B to always merge the debug branch from machine A when I pull:


232 git config branch.debug.remote origin
234 git config branch.master.remote origin
236 git config branch.master.remote origin/debug

All of these commands completed silently. None accomplished what I wanted. In the end I renamed master to old and debug to master to avoid having to fight it. Then I blew away my working copy and re-cloned because those config statements had created a new problem that I didn't know how to undo.

I'm sure the git virtuosos out there will know what was wrong. That's not the point. The point is that the tool gave me no feedback. It was like git was telling me, "Figure it out yourself. Or don't. I don't care." Which is par for the course with my git experience so far.

FormAlchemy 1.1: admin app, composite key support

2008-12-18T06:14:00.000-08:00

FormAlchemy 1.1 is out, so you no longer need to run trunk to get the admin app goodness -- now with i8n support. We also added support for all composite primary keys, and most composite foreign keys. (The distinction is, rendering an object depends on the PK, but loading relations depends on FKs.) Gael also added the fsblob extension, which allows storing blobs on the filesystem and the path in the database. (FormAlchemy can handle blob-in-the-db out of the box.)

(I previously blogged about basic FormAlchemy and the admin app, which are still good introductions.)

FormAlchemy has pretty good documentation. The most important page is form generation; instructions to configure the admin app are here.

An unusual approach to log parsing

2008-11-03T07:21:00.000-08:00

I saw an interesting article about logging today on reddit, and it struck a nerve with me, specifically how most text logs are not designed for easy parsing. (I don't agree with the second point, though -- sometimes logging is tracing, or perhaps more accurately, sometimes tracing is logging.)

We had a lot of log and trace data at Mozy, several GB per day. A traditional parsing approach would have been tedious and prone to regressions when the messages generated by the server changed. So Paul Cannon, a frustrated lisp programmer, designed a system where the log API looked something like this:

self.log('command_reply(%r, %r)' % (command, arg))

Then the log processor would define the vocabulary (command_reply, etc.) and, instead of parsing the log messages, eval them! This is an approach that wouldn't have occurred to me, nor would I have thought of using partial function application to simplify passing state (from the log processor and/or previous log entries) to these functions. (e.g., the entry for command_reply in the eval namespace might be 'command_reply': partial(self.command_reply, db_cursor, thread_id))

There are drawbacks to this approach; perhaps the largest is that this works best in homogeneous systems. Python's repr function (invoked by the %r formatting character) is great at taking care of any quoting issues necessary when dealing with Python primitives, as well as custom objects with some help from the programmer. But when we started having a C++ system also log messages to this system, it took them several tries to fix all the corner cases involved in generating messages that were valid python code.

On balance, I think this un-parsing approach was a huge win, and as the first application of "code is data" that made more than theoretical sense to me it was a real "eureka!" moment.

A small admin app for Pylons

2008-10-24T19:07:00.000-07:00

I said that it would be possible to build a django-style admin interface for Pylons using FormAlchemy. (That is, generate a UI for basic CRUD operations for all your models, with no further configuration necessary.) I have a proof of concept in FA svn; it's missing some obvious features like internationalization so there is no official release yet. But the basics are there, so in the meantime, if you'd like to kick the tires, just install FA from svn and give it a try.

Here are some screenshots from a pylons app incorporating models from the FA test suite. (The admin controller is fully customizable using standard FA (and Pylons) techniques, but these are what you'd see out-of-the-box.)

Index: Order page: Creating a new Order: Deleting an Order: The User page: Editing a User instance: Documentation on using and customizing the pylons admin app is here.

FormAlchemy 1.0

2008-10-16T20:35:00.000-07:00

A little background: a few months ago, I went looking for a web framework that was good at automating CRUD (create/retrieve/update/delete) against an existing database schema. I tried django but its database introspection abilities are beyond feeble, and django-sqlalchemy was not mature enough. I tried dbmechanic but its dozen-plus dependencies, most of which were alpha-quality, gave me pause; so did its basic architecture on top of toscawidgets, which I think is The Wrong Way to build web apps. (I understand that the former problem has since been reduced; the latter has not.)

So, I went back to option #3, FormAlchemy. I knew SQLAlchemy could reflect very hairy schemas indeed, and what it could not reflect, it could certainly represent with a little manual help. And FormAlchemy was a decent start to automating CRUD with SA models. I added the ability to represent relations, automatic syncing of form input back to SA objects, Grid support, and a test suite. Then Gael came along and added internationalization, support for even more SA features, and Sphinx docs. Along the way we've killed enough bugs and added enough test cases (yes, the two are related) that we think we have a pretty solid release. Especially since I just released 1.0.1 fixing the most obvious problems. :)

I think all three FA committers use it mostly with Pylons; that said, FormAlchemy has no dependencies besides SQLAlchemy itself. You could easily use it with werkzeug or web.py or whatever.

Here, finally, is a quick FormAlchemy tutorial:

To get started, you only need to know about two classes, FieldSet and Grid, and a handful of methods:

render: returns a string containing the html
validate: true if the form passes its validations; otherwise, false
sync: syncs the model instance that was bound to the input data

This introduction illustrates these three methods. For full details on customizing FieldSet behavior, see the documentation.

We'll start with two simple SQLAlchemy models with a one-to-many relationship (each User can have many Orders), and fetch an Order object to edit:

 from formalchemy.tests import Session, User, Order
session = Session()
order1 = session.query(Order).first()

Now, let's render a form to edit the order we've loaded.

 from formalchemy import FieldSet, Grid
fs = FieldSet(order1)
print fs.render()

This results in the following form elements:

Note how the options for the User input were automatically loaded from the database. str() is used on the User objects to get the option descriptions.

To edit a new object, bind your FieldSet to the class rather than a specific instance:

  fs = FieldSet(Order)

To edit multiple objects, bind them to a Grid instead:

 orders = session.query(Order).all()
g = Grid(Order, orders)
print g.render()

Which results in:

Saving changes is similarly easy. (Here we're using Pylons-style request.params(); adjust for your framework of choice as necessary):

 fs = FieldSet(order1, request.params())
if fs.validate():
    fs.sync()
    session.commit()

Grid works the same way. More details in the documentation; start with Form generation.

To give FormAlchemy a try, just easy_install it. If you have any questions, Alex and I are often in both #sqlalchemy and #pylons on freenode. And of course there's always the mailing list.

Available

2008-09-25T14:29:00.000-07:00

Feature50 is winding down now that CEO Ben Galbraith has accepted a job offer elsewhere. So, I'm interested in exploring my options, specifically, opportunities to build out the technology for a start-up working in concert with a strong business CEO. I've done this twice now.

Technical ability

I am a senior developer specializing in back-end technologies. At Mozy, where I was employee #2, I wrote a distributed file repository that stores petabytes of data, an amount comparable to Amazon's S3. I have 8 years of experience with PostgreSQL. I know how to design for scale, and how to find and remove bottlenecks. I am not afraid of diving into a new code base; I took over as maintainer of the Spyce web framework and the FormAlchemy toolkit, and I have contributed features or patches to SQLAlchemy, Pylons, and Jython, among others.

Soft skills

I enjoy building and working with a team. At Feature50 I am responsible for technical interviews, and personally recruited five of our first eight developers. At Mozy, I recruited three of the first five. I designed a customized version of Review Board -- a code review tool -- for Feature50 and MediaBank, and contributed several patches back to the project. I am active in the Python community and spoke at the last three PyCon conferences. I have spoken at OSCON and I am speaking at PostgreSQL Conference West in October.

The bottom line

I'm looking to work on a challenging project -- that is, not Yet Another CRUD App -- with a small team. I am currently based in Utah; I am willing to work remotely or relocate. Contact me at jonathan at utahpython dot org.

Blog Day recommendations

2008-09-01T02:14:00.000-07:00

As with many things in the blogging echo chamber, blog day takes itself a little too seriously. But it's impossible not to love an excuse to talk about some of my favorite blogs that don't seem to have as much exposure as I think they deserve:

Theo Schlossnagle, CEO of OmniTI, a scalability and performance consulting company. His best posts deal with scalability at the ops level. His book is good, too.
Greg Linden, ex-Amazon engineer, ex-Findory founder, current MS Live Labs employee. He likes to post analyses of interesting CS talks and papers, particularly in the area of collective intelligence. Greg stays very on-topic so the most recent posts are about as representative as any.
Chris Siebenmann writes about life as a professional sysadmin. He also sometimes blogs about python.
Josh Berkus, PostgreSQL core team member, mostly blogs about current events in the database world, but every once in a while he writes a must-read post about database design. Google thinks that "Rules for Database Contracting" is his most popular post, and that's a good pick too.
A non-technical pick: Eric Burns is the Gene Siskel of web comic critique. Except he's not dead.

App Engine conclusions

2008-08-29T14:16:00.000-07:00

Having been eyeball deep in App Engine for a while, evaluating it for a project at work and putting together a presentation for the utah open source conference, I've reluctantly concluded that I don't like it. I want to like it, since it's a great poster child for Python. And there are some bright spots, like the dirt-simple integration with google accounts. But it's so very very primitive in so many ways. Not just the missing features, or the "you can use any web framework you like, as long as it's django" attitude, but primarily a lot of the existing API is just so very primitive.

The DataStore in particular feels like a giant step backwards from using a traditional database with a sophisticated ORM. Sure, it can scale if you use it right, but do you really know what that entails?

Take the example of simple counting of objects. There's a count() method, but in practice, it's so slow you can't use it. Denormalize with a .count property? Yeah, that doesn't scale either: what you really need is a separate, sharded Counter class. And yes, sharding is very, very manual. (See slides 18-23 in the link there, and the associated video starting about 19:00.)

You can't perform joins in GQL. Or subselects. Or call functions, aggregate or otherwise. EVERYthing you are interested needs to be pre-computed. (Or computed by hand client-side, which is so slow it's barely an option at all.) I can extrapolate from this to my experience in production schemas and it's not pretty.

Of course, you also lose any ability to write declarative, set-based code, which is demonstrably less error-prone than the imperative alternative. Take a simple example from my demo app. Marking a group of todo items finished is four statements:

items = TodoItem.get_by_id(
  [int(id) for id in request.POST.getlist('item_id')])
for item in items:
  item.finished = datetime.now()
  item.put()

Compare this with SQL:

cursor.execute("update todo_items set finished = CURRENT_TIMESTAMP where id in %s",
             ([int(id) for id in request.POST.getlist('item_id')]))

Scalability is great but taking a big hit to back-end productivity is too high a price for all but a few applications. GAE is still young, so maybe Google will improve things, but their attitude so far seems to be "we know how to scale so shut up and do it the hard way." I hope I am wrong.

App Engine slides, code

2008-08-29T11:11:00.000-07:00

My App Engine 101 slides and code are up now.

Bad news: my macbook pro did not work with the projector, period.

Good news: I have seen it do this before (in a room with several mac experts -- it was not user error) and brought a backup laptop.

Bad news: I forgot to include the django beta1 framework in my code upload, so I told people to just download it. But beta2 was out, and didn't work with the version of App Engine Helper I had. (It looks like r58 fixes this.) Manual poking about the django download site ensued until I got a new zip uploaded.

Good news: the conference organizers liked it anyway and asked me to present a second time later in the day. Everything just worked the second time around.

Google App Engine at the Utah Open Source Conference

2008-08-25T16:18:00.001-07:00

App Engine is probably the biggest thing to happen to Python this year, so of course I volunteered to give a presentation on it at at the Utah Open Source Conference. (I'm scheduled for Friday, Aug 29, at 10:00 AM.) Last year's conference was a big success, so I'm looking forward to an even better experience this year.

Here's the abstract I submitted, before they blew away my paragraph breaks:

Google launched the App Engine service earlier this year to immense interest from the web development community. App Engine allows running applications on Google infrastructure, including BigTable, Google's non-relational, massively scalable database.
App Engine is appealing both at the low end, where small shops don't want to have to deal with hardware procurement and systems administration, and at the high end, where the kind of "instant scaling" App Engine promises to deal with bursty traffic is the holy grail of infrastructure planning. This tutorial will cover the basics of App Engine development, including development and deployment of a simple application.
Please sign up for an App Engine account and download the SDK ahead of time so we can jump right in to the code. Basic Python knowledge will be assumed.

After I submitted the proposal, I found out that all presentations are going to be 60 minutes long. That is not much time if we're going to do hands-on work, but you retain so much more by doing than you do merely from watching that I don't consider it optional. So seriously, come with the SDK installed. Those who do not, can look over the shoulders of those who do.

If you don't know Python and you're a last minute kind of person, you might want to attend Matt Harrison's talk the day before, 90% of the Python you need to know. Matt has presented several times at the Utah Python User Group as well as PyCon.

Bonus tip: if you can't make it to the UTOSC, the two best talks on App Engine are Rapid Development with Python, Django, and Google App Engine and Building Scalable Web Applications with Google App Engine. My presentation will cover similar material to the first of these.

A reminder

2008-08-15T14:47:00.000-07:00

Now that I've been doing Python full time again for a while it's easy to forget how magical it can be.

Last night I got an IM from a friend of a friend asking for (a) a recommendation for a Python book and (b) advice on writing a screen scraper. I pointed him to Dive Into Python and BeautifulSoup. Just now he IMed me again, "Hey, thanks for the tip. I ended up writing a screen scraper that I hadn't completed in 2 days in Groovy in about 20 minutes last night in Python with BeautifulSoup. So thanks, you got another python convert."

I love my job.

SQLAlchemy-Migrate for dummies

2008-07-22T16:06:00.000-07:00

I'm gave sqlalchemy-migrate a try today. I like it, and I'm going to keep using it. The one downside is that it's a bit hard to find "the least you need to know" in the documentation, especially if you lean old-school like me and prefer to write your upgrade scripts in raw sql. So here's my stab at it.

Create a "repository" for upgrade scripts:

migrate create path/to/upgradescripts "comment"

Create your manage script. If you have development/production dbs with different connection urls, create two scripts with the same repository but different urls:

migrate manage dbmanage.py --repository=path/to/upgradescripts --url=db-connection-url

For each database, create the Migrate metadata (a migrate_version table):

./dbmanage.py version_control

Create an upgrade script. This will create a script [next version number]-[database type]-upgrade.sql in the "versions" subdirectory of your "repository." That's all, so you could certainly do this by hand if you prefer, but letting the script do it is less error-prone:

./dbmanage.py script_sql sqlite

Edit the script.

For each database, apply the upgrade:

./dbmanage.py upgrade

Repeat the script/upgrade process as needed. That's it! Everything else is optional!

(What this gives you is a process where all your developers can have their own local database for development, and all they have to do is "svn up; ./dbmanage.py upgrade" without having to worry about which upgrade scripts have been applied or not.)

How to tell when you're successful

2008-07-13T21:53:00.000-07:00

You're successful when someone tries to get a cheap clone of your site done on a cheap-labor code monkey site.

I'm flattered, I think. (Although I'd be more flattered if it were a good code monkey site.)

Brief review of the Matias Half Keyboard et al

2008-06-25T00:12:00.000-07:00

I ended up buying four pieces of equipment to help deal with being temporarily one-handed: the Matias half keyboard, the X-keys foot pedal (cheaper than the Kinesis pedals, which got lukewarm reviews on Amazon), the Keyspan PR-US2 Presentation Remote, and the Pacific Outdoors 17-LC100 Folding Recliner.

The good: I'm very pleased with the recliner and modestly happy with the remote. I got the recliner to take naps in; the brace on my arm didn't really accomodate lying down. This $80 recliner compares well with zero gravity recliners costing over 10x as much. (I've used two of the expensive variety; a BackSaver and one whose brand I don't recall.) The only downside is you can either sit up, or recline fully; there is supposedly a way to adjust the recline angle, but it doesn't really work. Expensive zero gravity recliners can all reliably lock at any angle you like.

The remote mostly worked as a mouse substitute that I could use with my immobilized right hand, reducing the need to slow down my left hand even more by switching from keyboard to mouse and back. Unfortunately, the mouse control pad is not nearly as good as one of the IBM "pointing sticks;" it appears to have four control points, like an old Nintendo D-pad, which gives only 8 possible directions to move in. This and a poorly quantized pressure sensitivity sometimes made things frustrating. If I were to do this again I would try a handheld trackball instead, even though I could not find any wireless models.

The bad: the half keyboard did not help programming speed with one hand, and the foot pedal didn't improve things. I've returned both.

The half keyboard gives you the left hand side of the keyboard, which toggles to the right side when the space bar is held down. So "a" becomes ";", "f" becomes "j", snd so on. For alphabetical keys, I found that it was true that I did not have to re-learn to touch type; I did not have to look at the keyboard, although I did have to pause and think, "does this one require the space toggle or not." I got up to about 20 wpm before giving up, compared to 25 with one hand on a full keyboard. I think I could have easily doubled that to 40+ wpm with enough practice to eliminate that pause and recognize "runs" of letters that can be typed without releasing the space, like "you," without thinking. But that kind of investment wasn't worth it because of a serious flaw.

The half keyboard is really more like a "1/4 keyboard." It only gives you the alphabetical keys and a couple punctuation marks. No number keys with their !@#$ counterparts. No F keys. No arrow keys. On a mac, you can have cmd or control but not both.

To allow these keys to be typed, there is a "numeric toggle" key that switches to keypad mode, and two other modes that you access by hitting "shift shift" and "shift shift shift." Almost any line of code you might want to type is going to run into this. Typing [0] for instance is shift shift s numerictoggle b numerictoggle shift shift a. Even the symbol-averse Java will need parentheses for method calls, and yes, parens require mode switching too. (As do braces. Shudder!)

So I lost in the non-alphabetical and modifier access much more than I could see myself gaining on the pure alphabetical side.

Finally, the modifier keys were on the right hand side of the keyboard where they very difficult to combine with shift. I tried to ameliorate the modifier key problems with the X-Keys pedal, mapping the pedals to cmd/ctrl/option, but that didn't really work either. (The included ikeys software wouldn't work at all. At least ControllerMate worked in non-X applications, but since Wing is the only IDE that does locals completion well, using a non-X IDE temporarily was a non-starter. Locals completion is nice with two hands, but absolutely essential with one.) Note that this is more of an OS X issue than a problem with these pedals; apparently mapping pedals (x-keys or kinesis) to modifier keys works fine on windows.

So, the half-keyboard is not useful for programmers. If it (a) were wireless and (b) had a non-skid backing -- it slid all over the place because the back side was just smooth plastic -- I could see it being useful for heavy smartphone users. But it fails there too. Good luck with this one, Matias.

Postscript: I considered trying the Frogpad as well as the half keyboard, but with users reporting that they got "up to 20 wpm after 2 weeks," it didn't sound worth the trouble. So if I ever had to spend another three weeks one handed I am not sure what is left to try. Probably I would try to use ControllerMate (os x) or xmodmap (linux) to make make a "half keyboard" in software that didn't suck so much, as suggested by one of the commenters in my first post.

One-handed typing?

2008-05-31T19:19:00.000-07:00

I separated my right shoulder so that arm is going to be out of commission for a while. (I am right-handed.) I'm managing about 25 wpm with one hand, or about 1/4 my normal speed. This is frustrating. The Handkey Twiddler has been out of production for a while. The BAT is not OS X compatible. Anyone tried the Half Qwerty keyboard? Are there other good options for under, say, $300? (I found several very niche products for significantly more.)

I do plan to try voice recognition for email and IM but I can't see that working very well for code.

Jython Notes

2008-05-19T22:22:00.000-07:00

I've been getting back into the Jython codebase this last week. The last time I submitted a Jython patch was in the beginning of 2004, so it's been a while. Things have changed... Jython is finally requiring Java 5 for the next release, which means the usual improvements, but especially good use of annotations. Here's some notes from my puttering around (mostly dragging Jython's set module up to compatibility with CPython 2.5's):

Expect Eclipse to be slightly confused. (Lots of "errors.") This is normal. Use ant to build.
ant regrtest is handy. run it before you start making changes so you know what's already broken in trunk. (At least between releases, jython does not appear to be religious about "no tests shall fail." But as a new developer you should make "no additional tests should fail" your motto.)
Subjective impression: Jython re performace is a bit slow. Jython uses its own re implementation predating the Java regular expressions in jdk 1.4. But, the JRuby guys reported that the jdk implementation doesn't perform very well, so Jython hasn't been in a hurry to switch. The JRuby solution was to port the oniguruma re engine from C to Java. But, Ruby's strings are byte-based and mutable where Jython's are not, so using the JRuby engine isn't just a matter of dropping it in. Also, these string differences may be a source of the poor performance the ruby people saw, so independant testing is in order here.
All of the Derived classes (PySetDerived, PyLongDerived, etc.) just exist to let python code subclass builtin types. Those derived classes are generated by a .py script in src/templates
If you add a Java class that needs to be exposed to python using the @Expose annotations, you need to add the class name to CoreExposed.includes, or Jython will default to picking attributes via reflection and it usually guesses wrong.
Given a PyObject, you can (usually) easily instantiate another PyObject of the same class with pyobject.getType().__call__(). The only times this won't work is when your type's __new__ does something tricky, like how PyFrozenSet or PyTuple return a singleton for an empty frozenset or tuple.

Thanks to all the people in #jython who helped me out, especially Philip Jenvey!

Quick tip for debugging with Jython

2008-05-16T05:33:00.000-07:00

Currently, Jython ships with the pdb debugger module from Python 2.3. Unfortunately the 2.3 pdb is primitive even by command-line debugger standards. (For instance, if the program you are debugging throws an exception, it will take pdb down with it. Seriously. Did anyone actually use this thing?)

Fortunately all you have to do to get a much better experience is grab pdb.py, bdb.py, and cmd.py (for good measure) from a 2.5 CPython installation and run against that instead.

I've only tested this with Jython trunk but I think it should Just Work with the 2.2 release, too.

IDE update

2008-05-09T03:42:00.000-07:00

Last night the Utah Python User Group held an editor/IDE smackdown. I'm not going to write an exhaustive summary, but here are some highlights:

ViM's OmniComplete is actually pretty decent. Calltip support in the GUI is also good. (GUI? ViM? Yeah, weird.)
Emacs completion, from Rope, is also good. Emacs's refusal to make any concession to GUIs though keeps things clunky. Not that it isn't great that Everything Works over plain ssh; that's fine, but going through classic Emacs buffers for docstrings or completion means everything takes more keystrokes than it should while being less useful than having that information Always On.
Rope also gives Emacs refactoring support that works surprisingly well.
PyDev still sees a big win from the Eclipse platform. Specifically, even though Subclipse and Subversive are a bit weak compared to the gold standard (that would be TortoiseSVN), they are much better than what you get with Komodo or Wing. Now that I am on OS X (no Tortoise) this is a bigger issue for me than it used to be.
PyDev Extensions has refactoring support now, too.
Komodo has limited support for completion inside django templates. Which is impressive, since the commands allowed in django templates aren't really Python, which is to say that you can't just use the same completion support that you use for normal Python code.
Mako template support with completion, anyone?
The latest versions of Komodo and Wing both integrate unittest support. Wing also supports doctest out of the box. Meaning, you click a button, your tests run, you get a pretty summary with click-to-go-to-the-source-of-the-error support. This might get me to finally upgrade to Wing 3. It's not that "python test.py" is so hard, so much as I do it so often that even a little more convenience adds up.

I was surprised how well ViM and Emacs do with Python now. ViM's modern inline interface for code completion and Emacs's refactoring support are particularly nice. The IDEs still win on the I part (Integration), in particular debugging and (for Eclipse at least) svn support.

Update: Ryan McGuire blogged about his Emacs presentation in more detail.

Update 2: John Anderson blogged about setting up ViM

How to piss off your customers in two easy steps

2008-04-11T00:43:00.000-07:00

Don't communicate with them
Treat them like they owe you something

Google is off to a good (bad?) start with both of these in its management of the App Engine release.

Of the 120+ issues logged by beta testers, a few have been closed as wontfix or duplicate; most have no response at all from the App Engine team. I can't think of any other company that I've filed an issue with that took that long to get back to me. The good ones get back within hours.

The one exception I have seen is for the urllib issue, where gu...@python.org, presumably Guido, wrote

Providing a urllib replacement implemented on top of urlfetch shouldn't be particularly hard. If someone is willing to produce one, I'd be happy to review it and, if it passes muster, try to get it added.

Paraphrased: "maybe if you do our work for us we'll consider it."

WTF!

This isn't OSS, where "if you want something, do it yourself" is at least a semi-valid response. App Engine developers are all currently beta testing a product that Google hopes to eventually charge for. We're doing google a favor. (Context: the replacement Guido wants is a piece of code that will only ever be useful on app engine, and is something Google should have done in the first place instead of making urlfetch a public API. This is not code with a use case outside of App Engine.)

Maybe I'm over-sensitive, but this really rubs me the wrong way.

I hope Google can (a) put enough engineers on this that they can actually respond to issues, and maybe start closing some, and (b) remember that when you're selling a product, "why don't you fix it if it bothers you" is a poor response.

The business case for Google App Engine

2008-04-10T23:39:00.000-07:00

App Engine sure has caused a stir. Some of the competition is already scared, with reason.

But who is App Engine's real competition?

In a lot of ways, App Engine is in a class by itself. It competes on the high end with Amazon Web Services. But it also competes on the low end with every shared host out there. And thanks to the integration of Google authentication and the application directory you could also make a case that in an orthogonal way it competes with Facebook's application API.

At the low end, App Engine is a big deal for Python developers and anyone else who is allergic to PHP. Historically, you've really had to look hard for low end hosting that offered anything else. And as everyone who has given products away to colleges knows, Free is a fantastic hook to get developers to try out your platform. Once it's open for all, App Engine is going to become the preferred option for developers with the itch to write a toy or proof of concept and show it off to the world.

Less obviously (to developers, anyway), App Engine also a big deal for businesses that aren't quite big enough to hire a sysadmin, or who are big enough but still prefer not to deal with that complexity. (You thought hiring skilled developers is hard? If anything, hiring skilled sysadmins is harder.)

I suspect there are a substantial number of companies in the uncomfortable situation of really needing more performance than shared hosting offers, but not wanting the complexity of taking the next step, to dedicated servers with dedicated sysadmins.

Of course, given App Engine's constraints, porting such applications to it is only going to be an option in a few cases. The question is, are managers of new projects farsighted enough to see this problem coming and realize that app engine insures against it?

At the high end, AWS is the only real competition to App Engine, but as most observers have pointed out, they are different beasts. AWS offers far more flexibility, at the cost of far more hours from your ops department. (Although App Engine's datastore is a lot more sophisticated than the AWS SimpleDb, so the capabilities of AWS aren't a strict superset of App Engine's.) Contrary to the Joyent assertion linked earlier, it isn't necessarily stupid to trade flexibility for convenience. App Engine just works to an unprecedented degree in the field of high-end scalability.

As with anything this disruptive, there's been a certain amount of hysteria. Even people who should know better have repeated the idea that "nobody will want to acquire a product built on App Engine because you're locked in." This is stupid. Depending on a proprietary platform hasn't stopped products built on Oracle from being acquired, or products using AWS, or even products built on a proprietary UNIX. (Yes, those still exist.) Nobody will care if you build on App Engine, except maybe Microsoft and Yahoo. And even they can be pragmatic; Hotmail ran on BSD when Microsoft acquired them.

Lock-in is a real issue, but not because App Engine will keep you from being acquired, and not because Google will screw you once they have you in their clutches -- that would scare off new customers and thus be bad business. Lock-in is an issue because evolving requirements might make App Engine's confines less of a good fit than it started out. If you have to start adding servers at AWS or RackSpace to handle things you can't within App Engine, App Engine loses most of its value.

Language popularity, App Engine - style

2008-04-09T23:07:00.000-07:00

Just for fun, here's the number of stars (interested people) for the different language-support feature requests for Google App Engine:

Perl: 85
Java: 69
Ruby: 67
PHP: 23
C#: 11
jvm, not just java: 7
Common Lisp: 5

Update: Perl is stuffing the ballot box :)

Google App Engine: Return of the Unofficial Python Job Board Feed

2008-04-09T21:53:00.000-07:00

Over three years ago (!), I wrote a screen scraper to turn the Python Job Board into an RSS feed. It didn't make it across one of several server moves since then, but now I've ported it to Google's App Engine: the new unofficial python job board feed.

I'll be making a separate post on the Google App Engine business model and when it makes sense to consider the App Engine for a product. Here I'm going to talk about my technical impressions.

First, here's the source. Nothing fancy. The only App Engine-specific API used is urlfetch.

Unfortunately, even something this simple bumps up against some pretty rough edges in App Engine. It's going to be a while before this is ready for production use.

The big one is scheduled background tasks. (If you think this is important, star the issue rather than posting a "me too" comment.) Related is a task queue that would allow those scheduled tasks to easily be split into bite-size pieces, which is important for Google to allow scheduled tasks (a) without worrying about runaway processes while (b) still accomplishing an arbitrary amount of work.

If there were a scheduled task api, my feed generator could poll the python jobs site hourly or so, and store the results in the Datastore, instead of having a 1:1 ratio of feed requests to remote fetches.

While you can certainly create a cron job to fetch a certain url of your app periodically, and have that url run your "scheduled task," things get tricky quickly if your task needs to perform more work than it can accomplish in the small per-page time allocation it gets. Fortunately, I expect a scheduled task api from App Engine sooner rather than later -- Google wants to be your one stop shop, and for a large set of applications (every web app I have ever seen has had some scheduled task component) to have to rely on an external server to ping the app with this sort of workaround defeats that purpose completely.

Another rough edge is in the fetch api. Backend fetches like mine need a callback api so that a slow remote server doesn't cause the fetch to fail forever from being auto-cancelled prematurely. Of course, this won't be useful until scheduled tasks are available. I'm thinking ahead. :)

Finally, be aware that fatal errors are not logged by default. If you want to log fatal errors, you need to do it yourself. the main() function is a good place for this if you are rolling your own simple script like I am here.

My half-baked thoughts on Python web frameworks

2008-04-06T22:14:00.000-07:00

I have been lucky to be able to fill our recent open positions with people who know Python as well as Java so now we are up to half the (6 person) company in that category and preferring Python, and 2 of the others have played with Python and liked it at least well enough to not object. So the boss has conceded that it makes sense to go the Python route for our next project.

We're going to be doing a web, "next gen" version of our existing client-server project, which is mostly simple CRUD but does have 1000+ tables in its current incarnation. So we really need something that can autogenerate 90+% of the CRUD or we will go insane.

The trouble is, I still don't really like any of the Python web options 100%. (I like the web options in other languages less, but I'm a perfectionist.)

Django is well documented, its admin app is something everyone else envies, and newforms looks decent, but the ORM blows and I'm not fond of the template engine either. (Pre-emptive pedantry: yes, I know I can "import sqlalchemy." Please stop saying that like it means something; I'm not interested in defining models twice -- once for real work with SA, and once for interop with the rest of django.) Apparently django-sqlalchemy got far enough in PyCon sprints that it's kinda usable so working on that would be an option. Of course even then there is no guarantee the django core would accept it into mainline, and maintaining it as a "vendor branch" would proably suck. If django used a dscm like Mercurial I might be willing to do that, but svn is just too painful so that is a real risk.

I don't see a way to generate a page containing just a CRUD interface for table X with the django admin app. The admin app really is a monolithic application, not something you can easily re-use pieces of.

Regexps suck for url mapping.

Pylons is not well documented and after keeping an eye on this for something like 18 months I don't think this is a problem that will be solved, for whatever reasons. On the other hand, SA + mako is a very sane default, and both of those are well documented so it's really only core Pylons that suffers from doc crapitude, and core Pylons is fairly small. IRC responsiveness mitigates this further.

Pylons still doesn't have a good CRUD (or even high-level manual form generation) solution, which has bugged me for even longer than the docs. I can't fathom how people can tolerate writing this kind of boilerplate in 2008. Formalchemy gets about 30% of the way there. DBMechanic requires TG2 atm, although apparently hacking it to run on Pylons may not be too much effort; I would guess around 20% of the effort to get the django-sa project really usable.

TG2 is of course very bleeding edge and although I like genshi's syntax in theory, in practice XML templates irritate the hell out of me. (Very verbose, xinclude sucks compared to "inheritance," and incorporating rich dynamic content -- i.e., user-generated, like forum posts, that needs to include html tags -- is a PITA. Not to mention that having to write "a > b" when you mean "a > b" bugs me all out of proportion to the actual inconvenience it inflicts on me.) Still, better than the django templates.

I'm skeptical that TG2 is a big enough value add to want to add it (in its unfinished state) as a dependency vs rolling our own on Pylons. But DBMechanic does look like it could be exactly what I want in a CRUD generator.

web.py seems like more of a tech demo than a real product. I don't see any signs of a CRUD or form generator. reddit, probably the largest web.py site at least in terms of page views, moved to Pylons.

Zope 3 is alone in being really production ready without running from svn. Grok does do a good job of smashing zcml and z3c.form looks okay but lives up to the Zope reputation of complexity. (Field managers, widget managers -- are these the same things? -- widget modes, ...) AFAIK relational dbs are still second-class citizens in zope, and with all due respect to zodb it is no postgresql. OTOH there is z3c.sqlalchemy which gives me hope. Finally: you have to manually restart zope (per the Grok tutorial) after changing your .py files? Seriously?

Bottom line, Zope might actually be a decent option if we had a Zope expert on staff but we do not and I am not willing to tackle the learning curve alone.

Nevow: form handling is in flux. The new hotness is "pollenation forms," but that is svn-only and the api "will probably change."

Zope and Nevow both have their own xml-based templates predecessing but similar to genshi. Something like Nevow's Stan is obviously useful for programmatic template generation but it's not yet clear if that's going to be something we need. Probably only if we have to write our own form generator. If so, I suspect ripping a standalone Stan out of Nevow would be straightforward.

(Spyce of course never really got any traction to speak of. It's time for me to let it go quietly into the night and leverage someone else's framework.)

Conclusion: I think porting DBMechanic to Pylons is our best option. DBMechanic seems designed to be more flexible than the django admin app. Django would be my second choice.

Corrections? Thoughts?

Real Python IDEs

2008-04-02T09:05:00.000-07:00

After reading a blog post titled "The Abysmal State of Python IDEs" (which I won't link to because it's minformative, but it's easy to google by title), I wondered how the author managed to pick such a lousy group of IDEs to try. He tried "ActiveState" (does he mean PythonWin?), DrPython, SPE, and ScrIDE, only one of which is in the top 10 google hits for Python IDE.

The google top 10 include Eric, Wing IDE, Radio Userland, SPE, PyDev, and Komodo. The Yahoo and MSN top 10s are similar. Except for Radio Userland, this is a much better group to start with, and one that in fact does include what I think are the only 3 Python IDEs worth trying.

So how does a newbie end up picking such a lousy group of IDEs to try? The only likely possibility seems to be that he went to the top google hit, the python.org wiki page. Or possibly he went off of the top MSN hit, the c2 wiki Python IDE page. Both are (rather, were) heaping wads of products that mostly weren't IDEs at all, or were IDEs for other languages that happened to include Python syntax coloring.

Syntax coloring and maybe a Run button doesn't qualify you as a Python IDE in 2008, guys. (Sorry, IDLE.) Integrated means you need to integrate something nontrivial, preferably a debugger, although gui builders can also count.

So I organized the python.org IDE page by feature set and moved the non-IDEs to the Editors page, even if a pedant would note that they were IDEs, just not really for Python. That's not what 99.9% of people are looking for when they go to a Python IDE page, so let's be useful rather than pedantic. I also elided the non-IDEs from the c2 page.

Best new blog I discovered at PyCon [so far]

2008-03-15T03:57:00.000-07:00

I was talking to Adam Gomaa on Thursday when Ben Bangert stopped by us and told him he had an interesting blog. "If Ben says you have a good blog, I'll have to check it out," I told Adam. "That's not what I said," Ben corrected me. "I said interesting." But it is good, and I'm glad I found it.

And regarding Adam's post on declarative layers for SQLAlchemy, check out the new-in-SA 0.4.4 declarative plugin. It's almost exactly what Adam was looking for -- a little more verbose, in keeping with the "explicit is better than implicit" Python philosophy that SA shares, but creating your own superclass that creates a PK named "id" by default is just a few lines of code if that's what you prefer.

PyCon, Saturday and Sunday

2008-03-15T03:29:00.000-07:00

Saturday I'll be at the SQLAlchemy and State of PyPy talks. Then the board game BOF in the evening. In between, probably mostly the "hallway track."

Sunday I plan to attend "What Zope did wrong" and "Core Python Containers." (I'd also like to see the Wingware presentation in the 11:35 slot, but since I can only pick one, I guess I can just cross my fingers that this year's video recordings actually get published somewhere.)

Feel free to stop me and introduce yourself.

(If we met at a previous PyCon, I've changed my hair around a lot from year to year. The photo on this blog represents what I look like now. I should probably stick with this for a couple years so people can recognize me.)

Introverting

2008-03-13T17:51:00.000-07:00

I'm taking a break during the evening tutorial in my hotel room on the second floor where I can enjoy the pycon wireless signal (which seems to be working quite well now). My name is Jonathan, and I am an introvert [too]. I suspect a lot of PyCon attendees can empathize.

After the last tutorial session ends at 9:30, I'm planning to head down with my copy of Munchkin and see if anyone wants to start the board game social a day early.

Slides from Introduction to SQLAlchemy tutorial

2008-03-13T13:11:00.000-07:00

My slides from this morning are up: http://utahpython.org/jellis/sa-intro.pdf. The about 1/3 of the class did not have SA installed yet, and the network was down. Fortunately, Mike and Jason brought 5 flash drives and by the time we got to the first exercise everyone was up and running.

This was my third time doing a three-hour SQLAlchemy tutorial. Differences from (last time) include

updated for the 0.4 series
removed almost all the SQL-layer material
added a section on the new relation filtering api
Improved the parts of the Fundamentals sections that were poorly explained
added a short section on the new-in-0.4 transaction management.

There wasn't a wall clock in the tutorial room, so despite making an effort to be aware of time I went 10 minutes over. Sorry, guys. :)

Jason Kirtland will be posting the slides from the Advanced SQLAlchemy tutorial soon.

Pylons: first impressions

2008-03-07T15:23:00.000-08:00

A couple co-workers and I spent some time with Pylons yesterday, enough to get to where we started to feel productive, but not much more than that. I think there's value in a newbie's first impressions, so here are mine. I'm sure at least some of these are wrong.

Poor documentation of core Pylons (Mako and SQLAlchemy are fine -- thanks, Mike). I had to use the source several times. I'm still not really sure how Routes works, although I was mostly able to make it do what I wanted. The first tutorial overcomplicated things, showing how to configure things to handle semi-obscure requirements, without explaining those requirements or simpler alternatives.
Helpful community. I got most of the answers I needed pretty quickly in the #pylons freenode IRC channel.
Not much black magic: if you know Python you won't be struggling with weird Pylons-only concepts. It's all modules, classes, and dicts put together in an intuitive way (at least to my way of thinking).
SA (SQLAlchemy) is an amazing pleasure to use. (Okay, not just in Pylons, but I had to say it.) I have a slightly unusual schema -- the details are outside my scope here -- and SA's autoload handled it perfectly.
I wrote more CRUD boilerplate than I would have liked. There is no real alternative to Django's "admin app." DBMechanic looks like it's getting close, but it's TG-and-Genshi only for now. FormAlchemy is a partial solution (I did use it) but only does html generation; you'll still write boilerplate in your controllers.
Genshi appeals to me in theory but in practice its XML nature makes it feel clunky. XInclude as an alternative to template inheritance? 3 lines of xmlns per template? Mako has its own verbosity problems, e.g., having to do a def to pass a title to the parent template, but these aren't inherent to Mako's approach the way they are to Genshi's (we're XML, dammit), and Mike seems mildy interested in improving this specific example for the next release.
The pyfacebook tutorial is long on throwing wads of code at you and short on explaining what's actually going on. What does facebook.check_session() do? What does the facebook_middleware do? Why? Most facebook api tutorials have this same problem. Obviously I haven't written a better one, so call me a hypocrite, but tutorial authors, please explain the why and not just the what.

SQLAlchemy at Pycon 08

2008-02-09T08:21:00.000-08:00

SQLAlchemy will be well-represented this year with two tutorials and a talk.

I'll be the primary instructor for the Introduction to SQLAlchemy tutorial. I just updated the pycon page with the outline of what we'll cover. The slides will be pretty similar to last time, only with more time spent on a high-level intro to ORM (object-relational mapping) for people who have little exposure to that. And of course last year 0.4 was not out.

The SQLAlchemy documentation is thorough but a little intimidating. IMNSHO, the introduction tutorial is a great way to pick up the basics and get some practice, after which everything starts to make a lot more sense.

Mike Bayer, the author of SA, will be the primary instructor for the Advanced SQLAlchemy tutorial. Jason Kirtland, one of the most prolific SA hackers besides Mike himself, will also be teaching.

At the conference itself, Mike will be presenting Sqlalchemy 0.4 and beyond. To save you digging it out of the talks page, here's the summary:

At last year's Pycon, we introduced SQLAlchemy, the Database Toolkit for Python. This year, SQLAlchemy has gained new developers, a lot more users, and has now produced SQLAlchemy 0.4. The latest series of SQLAlchemy is significantly improved from the previous, in that APIs have been greatly pared down and refined, performance has been stepped up 30-40%, and ongoing architectural and developmental improvements have made room for lots of great new features with more to come. This talk intends to describe what's new in the 0.4 series, both for current users as well as for folks who may have only had experience with our earlier versions.

Why IE rejects your cookies for no apparent reason

2008-01-14T21:08:00.000-08:00

Seriously, WTF.

I'll summarize for those of you who are allergic to MSN knowledge base articles, although this one is fairly to-the-point:

If you implement a FRAMESET whose FRAMEs point to other Web sites on the networks of your partners or inside your network, but you use different top-level domain names... IE silently rejects cookies sent from third party sites.

This bit me today while adding facebook support to my text-based game -- I'm going the IFRAME route for fb support rather than rewrite the whole app in FBML thankyouverymuch, and yes, apparently IFRAME counts too for IE retard-mode.

What makes me cry a little inside is not the two hours spent deep in old and crufty login and cookie-setting legacy code wondering what the flaming hell was going on. No, what makes me cry is that I got screwed by a setting that will never block the bad guys, because labeling yourself a good guy is entirely voluntary. It's like someone at MS read the evil bit RFC and took it seriously.

The mind boggles.

In the meantime, if you know where your web framework's cookie code lives, do everyone a favor and patch it now to add that P3P header given in the knowledge base by default. And an option to disable it if you're obsessive-compulsive that way.

Scala: first impressions

2007-12-30T16:04:00.000-08:00

I'm reading the prerelease of the Scala book, since I'm working for a heavily Java-invested organization now and programming in Java feels like running a race with cement shoes. Politically, Jython doesn't seem like an option; Scala might be an easier sell.

Here's some of my impressions going through the book.

Scala is a decent scripting language

[updated this section thanks to comments from anonymous, Jörn, and Eric.]

Here's how you can loop through each line in a file: Python:

import sys
for line in open(sys.argv[0]):
   print line

Scala:

import scala.io.Source
Source.fromFile(args(0)).getLines.foreach(println)

The scala interpreter also has a REPL, which is nice for experimenting.

This article has more examples of basic scripting in scala.

Getters and setters

One thing I despise in Java is the huge amount of wasted lines of code dedicated to getFoo and setFoo methods. Sure, your IDE can autogenerate these for you, but it still takes up lines in your editor and effort to mentally block them out when examining an unfamiliar class to determine what it does.

C# has Python-style properties, so in theory could be virtually free of this kind of boilerplate, since there is no syntactic difference between "foo.x = y" whether x is a raw field or a property. So, the right thing to do, which you'll see in Python code, is using a raw public field until you actually need extra logic, at which point you replace it with a property and nobody's code breaks.

But C# wasn't far enough removed from Java culturally so everyone writes boilerplate properties instead of boilerplate getters and setters. (I'm aware that in a compiled language like C# it makes sense to start with properties for classes in certain kinds of libraries but these make up a vanishingly small number of actual classes in the wild.)

This is a long way of saying that Scala's properties make a lot of sense given the audience they are targetting (Java programmers). Scala vars are automatically turned into properties (or getters and setters, if you prefer to think in terms of those). So even the most obstinate fan of boilerplate code has absolutely no reason to keep writing unnecessary getters and setters.

If you need to manually tweak your properties later, you can magically define a setter for "field" by naming a method "field_=". This seems a bit like a one-off hack rather than part of a well-designed system, but I can live with it. (Since parentheses are optional for a scala method taking no arguments, any no-argument method call is already syntactically indistiguishable from a raw field read.)

Scala doesn't know what it wants to be when it grows up, yet

Scala contains a lot of features from a lot of different influences, which means there's often competing styles to use; the young scala community hasn't yet decided which to emphasize. For instance, iteration vs traversal -- "for (item <- items)" vs "items.foreach" -- or even whether to leave out optional (inferred) type declarations.

Perhaps this is merely a failure of one book trying too hard to make scala be all things to all readers.

Random thoughts

Scala needs something like Python's enumerate function; if you want to loop each object in a collection but you need its index too, you have to do a manual for loop.
Using parenthesis for collection access ("array(0); map("foo"))") makes it unnecessarily difficult to tell if you're looking at a method call or a collection access.
The scala stdlib is not very google-able yet; if you don't know exactly the class you are looking for ahead of time, you probably won't find it. For example, when looking for a scala object that could iterate through lines in a file, I correctly guessed the scala.io package, but would never have bothered looking at Source until more googling turned it up in a blog entry.

Conclusion and disclaimer

This is long enough; I'll probably write more as I go through more of the book. Patterns and implicit conversions are particularly interesting.

I was attracted to scala while looking for a sane alternative to Java on the JVM and it looks like scala might "drag Java programmers halfway to Python," to paraphrase Guy Steele. And I'm happy to be corrected on anything I've written here.

Troubleshooting the ps3 wireless network connection, including error 80130128

2007-12-28T19:41:00.000-08:00

My father got a ps3 for Christmas, but ran into some problems getting it on his wireless network. The first one was "connection error 80130128" after configuring it to use DHCP. I couldn't google anything useful about this; just a few other hapless victims asking if anyone had any ideas. Fortunately Dad had his laptop there too and noticed Windows complaining that two machines on the network were both using the same IP. So, over the phone, I walked him through setting up the ps3 with a static address:

on his laptop, run -> cmd
ipconfig
Read the "gateway" ip. Put that into his browser to go to his router's admin page
Find the DHCP settings for his router to see what range of IPs it hands out; pick one outside that range
Set up the ps3 with that IP, the router IP as primary dns, and an opendns server as secondary

This made the connection test happy. But when he tried to go to the playstation store, it gave a DNS error. If he repeated the connection test again, it failed too. "Well," I told him, "It's supposed to try both DNS servers. But we can try setting the primary DNS server to opendns as well." Once he did that, everything worked.

Reed-Solomon libraries

2007-11-16T16:32:00.000-08:00

If you want to run a multi-petabyte storage system then you don't want to do it with Raid 5 or Raid 6; with modern disks' ~3% per year failure rate, that's 300 a year when you have 10000 disks and the odds start to get pretty good (relatively speaking) that you'll face permanent data loss at some point when you lose a third disk from an array while two are rebuilding. And of course monitoring and replacing disks in lots of small arrays is manpower-intensive, which to investors translates as "expensive."

You probably don't want to go with triplication, either; disks are cheap, but not so cheap that you want to triple your hardware costs unnecessarily. While storing multiple copies of frequently used data is good, all your data probably isn't "frequently used."

What is the solution? As it turns out, Raid is actually a special case of Reed-Solomon encoding, which lets you specify any degree of redundancy you want. You can be safer than triplication with a fraction of the space needed.

I was prompted to write this because Mozy open-sourced the Reed-Solomon library I used while I was there, librs, complete with Python bindings. The original librs we used at Mozy was written by Byron Clark, a formidible task. Later we switched to the version you see on sourceforge, based on Plank's original encoder. I wasn't involved with librs at all except to fix a couple reference leaks in the Python wrapper.

But if you're actually looking for an rs library to use, Alen Peacock, who is much more knowledgeable than I about the gory details involved here, tells me that if you are starting from scratch the two libraries you should evaluate are zfec, which also comes with Python bindings, and Jerasure which is an updated -- i.e., probably faster than his first -- encoder by Plank. (Jerasure has nothing to do with Java.)

Utah Data Recovery

2007-10-18T08:15:00.000-07:00

About three years ago (so pre-Mozy and definitely pre-Mac Mozy) my brother had his powerbook hard disk die. As in, not just mostly dead -- it would not power up. It had a lot of stuff on it that he didn't want to lose, but he felt like the usual suspects who charge $1k to $2k for data recovery were a rip off. So he hung onto the disk in case a cheaper option came along.

Then just recently when I saw some people on a local linux group mailing list recommend utah data rescue I suggested to my brother that he give it a try. UTDR starts at "only" $300.

UTDR did indeed recover the data, although they charged $100 extra for this one. Mac fee? Tricky hw problem? I don't know. But it was still a lot cheaper than the other companies I googled for fixing a physically dead drive. (As opposed to a corrupt partition table or something where the hardware itself was okay.) At least, the ones that actually give you a price up front rather than hiding behind "request a quote!"

Semi-automatic software installation on HP-UX, with dependencies

2007-10-09T19:52:00.000-07:00

I had to install subversion on a couple HP-UX boxes. Fortunately, there's an HP-UX software archive out there with precompiled versions of lots of software. Unfortunately, dependency resolution is like the bad old days of 1997: entirely manual. And there's fifteen or so dependencies for subversion.

So, I wrote a script to parse the dependencies and download the packages automatically. It requires Python -- which you can install from the archive with just the Python package and the Db package -- and BeautifulSoup, which you can google for. Usage is

hpuxinstaller <archive package url> <package name>
[e.g., hpuxinstaller http://hpux.cs.utah.edu/hppd/hpux/Development/Tools/subversion-1.4.4/ subversion]
[wait for packages to download]
gunzip *.gz
[paste in conveniently given swinstall commands]

Here is the script:

#!/usr/local/bin/python

import urlparse, urllib2, sys, os
from subprocess import Popen, PIPE
from BeautifulSoup import BeautifulSoup

required = {}
if not os.path.exists('cache'):
    os.mkdir('cache')

def getcachedpage(url):
    fname = 'cache/' + url.replace('/', '-')
    try:
        page = file(fname).read()
    except IOError:
        print 'fetching ' + url
        page = urllib2.urlopen(url).read()
        file(fname, 'wb').write(page)
    return page

def dependencies(url):
    scheme, netloc, _, _, _, _ = urlparse.urlparse(url)
    soup = BeautifulSoup(getcachedpage(url))
    text = soup.find('td', text='Run-time dependencies:')
    if not text:
        return
    tr = text.parent.parent
    td = tr.findAll('td')[1]
    for a in td.findAll('a'):
        yield (a.contents[0], '%s://%s%s' % (scheme, netloc, a['href']))

def add(name, url):
    required[name] = url
    for depname, depurl in dependencies(url):
        if depname in required:
            continue
        print "%s requires %s" % (name, depname)
        required[depname] = depurl
        add(depname, depurl)
        
def download(full_url):
    print 'downloading ' + full_url
    _, _, path, _, _, _ = urlparse.urlparse(full_url)
    fname = os.path.basename(path)
    f = file(fname, 'wb')
    def chunkify_to_eof(stream, chunksize=64*1024):
        while True:
            data = stream.read(chunksize)
            if not data:
                break
            yield data
    for chunk in chunkify_to_eof(urllib2.urlopen(full_url)):
        f.write(chunk)


# Compute dependencies before checking for installed files, since swinstall
# can let a package be installed w/o its dependencies. If there are such
# packages installed we don't want to skip their [missing] dependencies.
add(sys.argv[2], sys.argv[1])

try:
    p = Popen(['swlist'], stdout=PIPE)
except:
    print 'Warning: unable to list installed packages'
    installed = {}
else:
    installed = set(line.strip().split()[0] for line in p.stdout if line.strip())

to_install = []
for name, url in required.iteritems():
    if name in installed:
        print name + ' is already installed'
        continue
    full_url = '%s%s-ia64-11.23.depot.gz' % (url.replace('/hppd/', '/ftp/'), url.split('/')[-2])
    to_install.append(os.path.basename(full_url))
    download(full_url)

if to_install:
    print "\nAfter gunzip, run:"
    for fname in to_install:
        print "swinstall -s %s/%s %s" % (os.getcwd(), fname[:-3], fname.split('-')[0])
else:
    print 'Everything is already installed'

Congratulations, Mozy

2007-10-05T16:41:00.000-07:00

I left backup service provider Mozy about three months ago, and yesterday they were acquired by EMC as rumored by techcrunch earlier.

The cool thing about startups is they pretty much have to hire people who are totally not qualified to do awesome things and let them try. There's no way Amazon would have hired me to write S3, but that's what I did for Mozy.

Mozy was the third startup I've been a part of, and the first to amount to anything. I was employee number #3 and saw it grow from sharing a single rented office to 50 employees in two years. With people who didn't think it was strange to wear a tie to work. Trippy.

Unfortunately I'm not there to witness the final stage of being assimilated by the Borg firsthand, but I hear that's not really any more fun than it sounds so perhaps it's just as well.

Nice work, guys.

Wing IDE 3, Wing IDE 101 released

2007-10-02T16:10:00.001-07:00

Wing IDE version 3 has been released.

The list of new features is a little underwhelming. Multi-threaded debugging and the unit testing tool (only supporting unittest -- does anyone still use that old module anymore?) are nice but I don't see myself paying to upgrade from 2.1 yet. Now if they could get the GUI to keep up with my typing in Windows, I'd pay for that... I guess this is a sign that Python IDEs are nearing maturity; Komodo 4 didn't have any earth-shaking new features either, at least as far as Python was concerned.

(Personally I think someone should start supporting django/genshi/mako templates already. Maybe in 3.1, guys?)

Following ActiveState's lead, Wingware has also released a completely free version, Wing IDE 101. The main difference is that where the most essential feature Komodo Edit leaves out as an incentive to upgrade is debugging, Wing IDE 101 includes the debugger but omits code completion. Wingware also continues to offer the low-cost Personal edition.

But the really big difference between Wing IDE 101 and Komodo Edit is that you can freely use Komodo Edit for paying work. Wing IDE 101, like Wing IDE Personal has a no-commercial-use clause. (Komodo versions compared; Wing versions compared.) I'm still of the opinion that at $180, Wing Professional will pay for itself in short order, but for the hobbyist, Komodo Edit is very compelling. I've been using it myself for TCL and XML editing for several months now and it's a nice little IDE.

Too bad Komodo's emacs bindings continue to suck balls -- I mean, it's one thing to not implement fancy things like a minibuffer or kill ring, but if you can't even get C-W (cut) right, there's not much hope. Users contributed much-improved Emacs bindings to the ActiveState bug tracker way back in the version 3 timeframe. I guess ActiveState just doesn't care.

That wasn't the pigeonhole I expected

2007-09-21T22:03:00.000-07:00

I went to the BYU CS alumni dinner tonight. At one point they briefly put everyone's name and position on a projector, one at a time. (At five seconds apiece it wasn't as tedious as it sounds.)

When it was my turn, it announced "Jonathan Ellis, System Administrator."

What the hell?

It turns out that when I RSVP'd I said I was a "python kung-fu master & sysadmin of last resort." (In the sense that, if you really can't find a better sysadmin, I know enough to be dangerous.)

Don't bother trying to be clever around bureaucrats.

Utah Open Source Conference 2007

2007-09-08T18:18:00.000-07:00

The first Utah Open Source Conference finished today. I heard that they had close to 300 attendees -- not bad at all for a freshman effort.

I reprised presentations that I've given before, on SQLAlchemy and distributed source control. My slides are on the presentations page (although if you've seen my slides from either before, there's not much new there -- I got lucky, SA 0.4 isn't stable yet so I stuck with 0.3.10).

I had to work Friday so I missed a lot of presentations, but of the one I saw my favorite was on Ganglia, which I hadn't heard of before but which looks quite useful for anyone running a bunch of servers that takes uptime and qos seriously. (This was actually Brad Nicholes's third presentation of the conference -- he must have been busy!)

Afterwards I went to the board games BoF and played Mag Blast. Fun little game.

What it means to "know Python"

2007-09-05T18:55:00.000-07:00

Since Adam Barr replied to my post on his book, I'd like to elaborate a little on what I said.

Adam wrote,

[F]or me, "knowing" Python means you understand how slices work, the difference between a list and a tuple, the syntax for defining a dictionary, that indenting thing you do for blocks, and all that. It's not about knowing that there is a sort() function.

In Python, reinventing sort and split is like a C programmer starting a project by writing his own malloc. It just isn't something you see very often. Similarly, I just don't think you can credibly argue that a C programmer who doesn't know how to use malloc really knows C. At some level, libraries do matter.

On the other hand, I wouldn't claim that you must know all eleventy jillion methods that the Java library exposes in one way or another to say you know Java.

What is the middle ground here?

I think the answer is something along the lines of, "you have to get enough practice actually using the language to be able to write idiomatic code." That's necessarily going to involve picking up some library knowledge along the way.

This made me think. What are the most commonly used Python modules? I decided to scan the Python Cookbook's code base and find out. This is a fairly large sample (over 2000 recipes), and further attractive in that most of the scripts there are reasonably standalone, so they're not filled with importing lots of non-standard modules. The downside is there is code dating back at least to the very ancient Python 1.5 version.

In 2000+ source files and almost 4000 imports of stdlib modules, here are the frequency counts of imported modules.

Is this a reasonable list? I obviously think I qualify as knowing Python well enough to blog about it. Of the modules above the 80% line, _winreg, win32con, and win32api are platform-specific; new is deprecated, string isn't officially deprecated but should be, and __future__ isn't really a module per se. I believe I've used all of the rest but xmlrpclib at some point, although my line of comfort-without-docs would be only about the 60% mark. I think anyone who programs professionally will quickly get to knowing well at least the modules up to the 50% line.

sys	473
os	302
24%
time	210
re	145
35%
string	140
random	103
threading	66
socket	57
os.path	52
types	50
Tkinter	47
50%
math	43
win32com.client	42
__future__	41
traceback	40
itertools	38
doctest	37
urllib	35
cStringIO	33
struct	32
60%
win32api	31
getopt	29
thread	29
ctypes	28
StringIO	28
inspect	26
win32con	25
copy	25
cPickle	25
operator	24
datetime	23
cgi	22
70%
Queue	22
urllib2	20
md5	20
base64	20
xmlrpclib	19
sets	19
optparse	19
logging	18
weakref	18
shutil	17
unittest	17
pprint	16
urlparse	15
getpass	15
httplib	15
pickle	15
_winreg	14
UserDict	13
signal	13
80%

For those interested, a tarball of the recipes I scanned is here, so you don't need to scrape the Cookbook site yourself. The import scanning code is simple enough:

import os, re, compiler
from collections import defaultdict

# define an AST visitor that only cares about "import" and "from [x import y]" nodes
count_by_module = defaultdict(lambda: 0)
class ImportVisitor:
    def visitImport(self, t):
        for m in t.names:
            if not isinstance(m, basestring):
                m = m[0] # strip off "as" part
            count_by_module[m] += 1
    def visitFrom(self, t):
        count_by_module[t.modname] += 1

# parse
for fname in os.listdir('recipes'):
    try:
        ast = compiler.parseFile('recipes/%s' % fname)
    except SyntaxError:
        continue
    compiler.walk(ast, ImportVisitor())
    print 'parsed ' + fname

# some raw stats, for posterity
counts = count_by_module.items()
total = sum(n for module, n in counts)
print '%d/%d total/unique imports' % (total, len(counts))

# strip out non-stdlib modules
for module in count_by_module.keys():
    try:
        __import__(module)
    except (ImportError, ValueError):
        del count_by_module[module]
        
# post-stripped stats
counts = count_by_module.items()
total = sum(n for module, n in counts)
print '%d/%d total/unique imports in stdlib' % (total, len(counts))
counts.sort(key=lambda (module, n): n)

# results
subtotal = 0
for module, n in reversed(counts):
    subtotal += n
    print '%s\t%d' % (module, n)
    print '%f' % (float(subtotal) / total)

Merging two subversion repositories

2007-09-04T07:22:00.000-07:00

Update: an anonymous commenter pointed out that yes, there is a (much!) better way to do this with svnadmin load --parent-dir, which is covered in the docs under "repository migration." All I can say in my defense is that it wasn't something google thought pertinent. So, for google's benefit: how to merge subversion repositories. Thanks for the pointer, anonymous!

I needed to merge team A's svn repository into team B's. I wanted to preserve the history of team A's commits as much as reasonably possible. I would have thought that someone had written a tool to do this, but I couldn't find one, so I wrote this. (Of course, now that I'm posting this, I fully expect someone to point me to a better, pre-existing implementation that I missed.)

The approach is to take a working copy of repository B, add a directory for A's code, and for each revision in A's repository, apply that to the working copy and commit it. This would be easy if svn merge would allow applying diffs from repository A into a working copy of repository B, but it does not. I can't think of a technical reason for this. (In fact, I seem to remember that early versions of the svn client did allow this, with dire warnings, but I could be mistaken and I don't have a 1.1 client around anymore.)

So I tried instead to use "svn diff |patch -p0", which worked great up until the first commit with a binary file. Oops. For the final version I ended up having to create a working copy for A, update to each revision there, then rsync to the right point in working copy B and call the "svnaddremove" script to mark files added or deleted. (This is suboptimal since we can get the exact changed paths from svn, and just copy those files over, but rsync is fast enough as long as your working copies stay in cache. The update and commit steps both consistently took longer than rsync in my timing.)

My script does not try to be intelligent about copies or moves that svn knows about. Team A did not use branches or tags much so I didn't put the effort in to deal with those the "right" way (which would be to also issue a cp/mv on B's working copy to preserve history). It also uses unix users to commit revisions with the same name as the original commit. Doing this obviously requires at least access to the repository server to add the right users. I used "svn log -q |grep ^r |awk '{print $3}' |sort |uniq |useradd."

Final note: the perl script in svnaddremove is a long way of writing "awk {print $2}", except that it preserves filenames with spaces in them. There is probably a much more clever way of doing this, too.

Here, then, is the merge script:

#!/usr/bin/python

# usage: svnimport <source wc path> <target wc path> <revstart> <revend>
# e.g. svnimport liberte-source trunk/liberte 1 2000

from subprocess import Popen, PIPE
try:
    from xml.etree import cElementTree as ET
except ImportError:
    from elementtree import ElementTree as ET
import sys, time

def system(*args):
    p = Popen(args, stdout=PIPE, stderr=PIPE)
    out, err = p.communicate()
    if err:
        raise err
    return out

# super-minimal log scraper
# for a better one see hgsvn's svnclient.py, http://cheeseshop.python.org/pypi/hgsvn
def parse_date(svn_date):
    date = svn_date.split('.', 2)[0]
    return time.strftime("%Y-%m-%d", time.strptime(date, "%Y-%m-%dT%H:%M:%S"))
def parse_svn_log_xml(xml):
    tree = ET.fromstring(xml)
    for entry in tree.findall('logentry'):
        d = {}
        d['revision'] = int(entry.get('revision'))
        author = entry.find('author')
        d['author'] = author is not None and author.text or None
        d['message'] = entry.find('msg').text or ""
        d['date'] = parse_date(entry.find('date').text)
        yield d
        
def edited_message(entry):
    msg = entry['message'].strip().replace('\r\n', '\n')
    addendum = '[original revision %s committed %s]' % (entry['revision'], entry['date'])
    if msg:
        return msg + '\n' + addendum
    return addendum

sourcepath, targetpath, revstart, revstop = sys.argv[1:]
# rsync foo bar and rsync foo/ bar/ are very different!
if not sourcepath.endswith('/'):
    sourcepath += '/'
if not targetpath.endswith('/'):
    targetpath += '/'

xml = system('svn', 'log', sourcepath, '--xml', '-r', '%s:%s' % (int(revstart) + 1, revstop))
for entry in parse_svn_log_xml(xml):
    revno = entry['revision']
    print 'merging revision %d by %s' % (revno, entry['author'])
    # merge in the revision
    system('svn', 'up', '-r', str(revno), sourcepath)
    print '\trsync'
    system('rsync', '-a', '--exclude=.svn', '--delete', sourcepath, targetpath)
    system('/tmp/svnaddremove', targetpath) # svn should add this.  hg already did.
    # commit as the correct author, if available
    author = entry['author']
    print '\tchown'
    system('chown', '-R', author, targetpath)
    quoted_message = edited_message(entry).replace('"', "'")
    print '\tci'
    system('su', author, '-c', 'svn ci %s -m "%s"' % (targetpath, quoted_message))

And here is svnaddremove:

#!/bin/bash

# odd, xargs is invoking svn add/rm w/ no args when grep returns no lines.
# fix that with the ifs.
# (don't use grep -q or svn gets pissed about broken pipe.)

if svn st $1 | grep ^\? > /dev/null; then
  svn st $1 | perl -ne 'chomp; @Fld = split(q{ }, $_, -1); if (/^\?/) { shift @Fld; print join(q{ }, @Fld) . "\n"; }' | xargs -n 1 -i svn add "{}"
fi
if svn st $1 | grep ^\! > /dev/null; then
  svn st $1 | perl -ne 'chomp; @Fld = split(q{ }, $_, -1); if (/^\!/) { shift @Fld; print join(q{ }, @Fld) . "\n"; }' | xargs -n 1 -i svn rm "{}"
fi

A brief reaction to "Find the Bug"

2007-07-30T18:51:00.000-07:00

I picked up a copy of Adam Barr's Find the Bug, which is a cool concept for a book. (5 languages, 50 programs, 50 bugs; see if you can spot them.)

I found the bug in the first program, in C, then skipped to the Python chapter. The first two programs were not too bad, as pedagogical exercises go (although iterating through substrings instead of a.startswith(b) in the 2nd was painful). The third, though, was "Alphabetize words," 25 sloc to perform the equivalent of

def alphabetize(buffer):
  L = buffer.split(' ')
  L.sort()
  return L

... doing everything about the hardest way possible.

Now, it's pretty hard to introduce a non-obvious bug into my version of this function, so it wouldn't be appropriate for Mr. Barr's book when written this way. But the right thing to do is to make the task more difficult, not dumb Python down to the level of C! It's very very painful to read Python written like that.

(Actually it's painful to read any language written at such a low level of expressivity, which is why I prefer not to use languages that really can't do any better.)

Final version of OSCON SQLAlchemy slides

2007-07-23T22:18:00.000-07:00

http://utahpython.org/jellis/sa-tutorial-oscon.pdf

Also the code snippets:

This is what I'll be using in my tutorial tomorrow.

Update: I forgot to "svn up" on my web server. So now the final version is up.

PEP rss feed is live

2007-07-09T12:28:00.000-07:00

After I complained that python.org could use a PEP rss feed, David Goodger invited me to volunteer to write one. So I did. (With Martin v. Löwis doing the integration with the site build script. Thanks Martin!)

The feed is live at http://www.python.org/dev/peps/peps.rss.

Opera 9.2 is a pretty good browser

2007-06-16T05:45:00.000-07:00

I've been trying Opera 9.2 for a week, and I'm pleased with it enough that it's going to continue to be my main browser. The main selling points for me are

MDI weirdness is mostly hidden now, I hated earlier Opera UIs
20-30% less memory use; even after poking about in the guts of about:config to force FF's memory cache to the same 10MB that I gave Opera (which exposes this option right in the UI), Opera consistently uses less memory for the same workload. (Without adding this option to FF, it would max out around 400MB instead of 150MB.)
feels snappier; opera seems quicker to start rendering something useful on slow-loading sites like 1up.com, although total render time is about the same. It's also instantaneous to open a new tab, which consistently takes around 1s on FF after I've been using it a while. I open and close tabs frequently.
UI takes up less space: I know it's possible to re-skin FF, but I'd have to google it to find out how. Opera makes it easy. I'm using the Fresh skin for Opera which condenses the File menu and nav bar to about half as much height as FF uses.
built-in AutoFill, so I save even more space by not needing Google Toolbar
javascript/DOM support is finally close enough to FF that most sites don't have to specifically code for Opera. Heavy ajax use is an exception of course; I still have to use FF for Google Docs. (Gmail and Maps work fine though, probably due to some effort by Google.)
Download manager goes in another tab by default instead of a separate window. I didn't realize how much FF's behavior annoyed me before.

Downsides:

Occasional rendering problems, such as when customizing a laptop on hp.com. Blogger.com doesn't redirect to my dashboard when I'm already logged in to my google account.
Very very slow navigating large pages, such as a slashdot comment thread with 400 comments. Isearch is even slower on large pages and can lock up the UI for minutes if you invoke it injudiciously.
Sometimes ignores a site's instructions to not cache a dynamic page
hard to tell which tab is active in the default theme. (Fresh fixes this.)

A workaround for the sys.excepthook bug

2007-06-14T18:59:00.000-07:00

About two years ago I reported the bug sys.excepthook doesn't work in threads. Then just recently someone asked in #utahpython if I had a workaround. Here it is (also added as a comment to the bug report) -- all we do is monkeypatch Thread.run to run the excepthook manually if there is an uncaught exception:

def install_thread_excepthook():
    """
    Workaround for sys.excepthook thread bug
    (https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230540&group_id=5470).
    Call once from __main__ before creating any threads.
    If using psyco, call psyco.cannotcompile(threading.Thread.run)
    since this replaces a new-style class method.
    """
    import sys, threading
    run_old = threading.Thread.run
    def run(*args, **kwargs):
        try:
            run_old(*args, **kwargs)
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            sys.excepthook(*sys.exc_info())
    threading.Thread.run = run

How DOS 1.0 cost me an hour of scratching my head

2007-05-31T17:13:00.000-07:00

A couple months ago, I migrated my text rpg Carnage Blender to a new server, with Ubuntu 6.06 on the new box.

For an unknown reason, ftstrpnm on the new box wouldn't generate the pngs I used in my captchas. It was easier to just check in the images from the old machine into the my svn repository than debug this, so I did.

The downside was that my working copy on my Windows laptop stopped being able to update from the repository. It would get to "words/con.png," and error out. Google, for once, didn't turn up anything useful.

Today I got motivated. I tried all kinds of ways to get this to work. A new checkout had the same problem on Windows, but on Linux worked fine. The svn command line client for windows didn't work any better than Tortoise -- instead of "Error: Can't open file '...words\.svn\text-base\con.png.svn-base': Access is denied", it barfed con.png to stdout, and died. This was a clue, but I didn't realize that until later.

Puzzled, I tried scp-ing con.png directly. No dice. Maybe it was a problem with my ssh server or client instead of subversion. So I tarred up my Linux working copy and untarred on Windows. Still it crapped out on con.png. I gzipped just con.png and tried to scp that over. That didn't work either.

I started experimenting with the filename itself. I could scp the .gz just fine if I renamed it to c.png.gz first. But "touch con.png" on windows failed, as did "touch con.txt." Finally I googled [windows filenames con] and found the answer at the top of the results, unfortunately from before I started reading oldnewthing or it might have jogged my memory.

It's time for python development to open up a little

2007-05-31T04:54:00.000-07:00

I found out from Brett Cannon's blog that an abstract base clase (ABC) PEP has been accepted.

I don't like this PEP. It's a very big (and more important, inelegant) change to Python's style. But my real complaint is that as big as this change is, and as much as I try to stay current with Python (subscribing to 30+ blogs) I didn't have a chance to get involved in the discussion until after the PEP was already approved.

Python is big enough now that there should be some mechanism for feedback from the community before the priesthood of python-dev writes something in stone. Currently, if you want to know about PEPs before they are approved, you have to subscribe to both python-dev and python-3000 (which isn't linked from either the mailing lists page or the dev page, btw). I really don't care about the vast majority of these lists' traffic but PEPs, at least some of them, are important.

If the python-dev summaries ever got updated this might be a potential solution, but even at their best I don't remember them ever getting closer than a month behind or so. And two weeks is probably too coarse-grained anyway.

I think what python.org really needs is a PEP rss feed. A friend thought that they already had one, but neither he nor I could find it. So if it exists, it's well-hidden. If it doesn't exist, it should. Please?

(And if it's easier for whoever's in charge of such things to give me access to the server and repository than to do it himself, then yes, I'm volunteering.)

Best Python book for beginners

2007-04-18T19:27:00.000-07:00

It's really surprisingly difficult for someone who has been programming for a long time to write about programming at a level appropriate for real beginners. The first time I taught a class full of beginners at Neumont, I tried to take things as slow as possible. Then I spent the next week covering the material from the first day even slower.

So when the UGIC asked me to recommend a book to get for the participants in the Introduction to Python, I looked at all the ones I could find, but they all either assumed too much existing knowledge or covered material that would just confuse a beginner. Often both. But then Michael Bernstein pointed me to Python for Dummies.

If you're looking to teach beginners, or you're a beginner yourself, Python for Dummies is by far the best option. There's a few sections that are strikingly inappropriate for a book at its level (new-style classes!?) but it's still much, much better than any of the other books on the market in this respect. As a bonus, it's also one of the few that covers Python 2.5.

Introduction to Python slides

2007-04-18T10:51:00.000-07:00

Here are the slides from my introduction to python at the UGIC conference today.

This presentation was meant for people with little to no programming experience. So I deliberately kept it pretty basic, and in fact in 90 minutes we only covered up to about slide 20 in the pdf. I also added an exercise before moving on to slide 10. ("Read 3 integers into a list, and print the sum.")

There were 17? people there (which was the room's capacity), so it was very nice to have Kevin Bell also answering questions individually during the exercises.

Mercurial presentation slides

2007-04-16T13:19:00.000-07:00

Thursday I presented on distributed source control and Mercurial to the utah python ug. Here are my slides.

Then on Friday, Mozilla announced that they're moving from CVS to Mercurial, joining OpenSolaris and Xen and others on hg.

It's exciting to see what is still a small and elegant tool gain traction like this, even though in some ways hg (and dscm in general really) is still in the early adopter stage.

Mozy code deathmatch

2007-04-10T14:26:00.000-07:00

My employer, the creator of Mozy, is running a programming contest this Saturday. 9 languages are allowed. The first 2 rounds are online; the finals are in American Fork (Utah), but if you make it that far you're guaranteed to win some money.

(We did this last year too; this year the prize money is doubled to $20k. Not to mention how we are super-experienced contest organizers now!)

New mailing list for utah python user group

2007-04-02T09:22:00.000-07:00

Since I neglected to archive the old list when moving utahpython.org to a new server, the utah python user group has a new mailing list courtesy of Google Groups. (At least this way we're not dependent anymore on my incompetent sysadminning.)

One thing I don't hate about Python

2007-03-29T22:23:00.000-07:00

Sure, some things about Python bug me. But that's not what this is about. I wanted to react to Jacob Kaplan-Moss's gripes instead of promulgating my own. Specifically, his problem with Python's interfaces, or lack thereof.

I think I can keep this brief: interfaces are a hack that Java uses because Gosling et al thought multiple inheritance was too confusing and/or dangerous. (I believe I've read something recently where Gosling said that this was one decision he might do differently if he were re-designing Java now with the benefit of hindsight, but I can't find the source. Anyone remember seeing that?)

Python has MI. It doesn't need interfaces. I'm a little baffled that someone on the django core team would cite this as a problem with Python.

Jacob's precise objection is,

I shouldn’t need to care care about the difference between something that pretends to be a list and something that really is a list.

That's just it! You don't! But of course what Jacob really means is, "It should be easy to discover what methods a library expects to find on MY object that pretends to be a list." Which seems reasonable. And sure, good documentation is always welcome.

But when you cross the line to an Interface, at least the kind of Interface where Python itself would raise an error if I ignored the recommendation and left a method out (because I knew it wasn't necessary), that's bondage & discipline. That's not Python.

Introduction to Python at UGIC conference

2007-03-02T12:07:00.000-08:00

I'll be giving a (very!) introductory Python workshop at the Utah Geographic Information Council conference in April. After my 90 minutes, Kevin Bell -- also of the utah python user group -- will present on specific GIS applications.

(Apparently Python is particularly big in GIS these days because one of the big vendors, ERSI, takes Python pretty seriously.)

PyCon web frameworks panel notes

2007-02-24T06:25:00.000-08:00

I represented Spyce for the web frameworks panel. It was pretty cool looking out at the standing-room-only crowd, even though let's face it, most people were not there because of Spyce. :)

James Bennett and Matt Harrison have notes posted online.

PyCon SqlSoup slides

2007-02-23T15:27:00.000-08:00

I uploaded my slides for my Sunday SqlSoup talk, so they're linked in the schedule now. I also uploaded them here.

Update: the profile.py module I showed is here.

PyCon open-space talk on Spyce after the web frameworks panel

2007-02-23T09:27:00.000-08:00

I'll demonstrate writing a simple app with Spyce, including the most painless Ajax you have ever seen. Come check it out at 3:40 in the Bent Tree II room.

PyCon SQLAlchemy tutorial slides

2007-02-22T16:35:00.000-08:00

My SQLAlchemy tutorial went pretty well for the most part. It was a fast pace but most people kept up pretty well. If I did it again I would add more of an intro to ORM in general for people who had never used one, but over half the attendees had used SO or django's or tried SA already. I would also paste more code from my slides into the samples download to save people typing during the exercises (I had some, but I would do more next time).

I think most people liked it; the main exception was one fellow who was in way way over his head and visibly pissed about it. (I used a list comprehension at one point and he had no idea what it was.)

The slides are here. (The .py files referred to in the slides have also been moved to the jellis/ subdirectory.)

Spyce at PyCon

2007-02-20T10:18:00.000-08:00

I'll be representing Spyce as a late addition to the Web Frameworks panel. I'm also planning a lightning talk on Ajax in Spyce 2.2 (which will be released as soon as I finish getting the docs in shape) and an open-space Introduction to Spyce.

See you there!

Best Google Tech Talks?

2007-02-15T18:03:00.000-08:00

My wife got me a PSP for Valentine's Day, so I'm looking for videos to put on it. Since Google makes it so easy to get a PSP-compatible version, I thought I'd start with theirs... Recommendations?

SQLAlchemy slides

2007-02-14T13:01:00.000-08:00

I presented on SQLAlchemy at the Utah python user group last Thursday; slides are linked here.

In retrospect, for a shorter presentation like this I should probably spend more time talking about the ORM features, and less about the SQL layer. Although the SQL layer is useful on its own, and essential for doing advanced mapping, I don't think it has the sex appeal that the ORM has.

(Although I do think the first part, about why ORMs should allow you to take advantage of your database's strengths rather than being limited to a MySQL 3 feature set, was useful.)

Komodo 4 released; new free version

2007-01-25T11:25:00.000-08:00

ActiveState has released Komodo IDE 4. Perhaps more interesting, if you're not already a Komodo user, is the release of Komodo Edit, which is very similar to the old Komodo IDE Personal edition, only instead of costing around $30, Komodo Edit is free. The mental difference between "free" and "$30" is much more than the relatively small amount of money; it will be interesting to see what happens in the IDE space now.

After a brief evaluation I would say Edit is perhaps the strongest contender for "best free python IDE." The only serious alternative is PyDev, which on its Eclipse foundation provides features like svn integration that Edit doesn't. PyDev also includes a debugger, another feature ActiveState would like to see you upgrade to the full IDE for. But Komodo is stronger in other areas such as call tips and, well, not being based on Eclipse. I also think its code completion is better, although this impression is preliminary.

It's also worth noting that so far, Edit doesn't sport the "Non-commercial and educational use only" restrictions that Komodo Personal had.

PyCon SQLAlchemy tutorial full

2007-01-22T15:09:00.000-08:00

See you there!

Caution: upgrading to new version of blogger may increase spam

2007-01-17T17:55:00.000-08:00

I was pretty happy with the old version of blogger, but I upgraded today so I can use the new API against my own blog. So far I have 4 spam comments (captcha is still on) versus about that number for the entire life of my blog under the old blogger. Bleh.

Could just be a coincidence. I hope so.

(Update Feb 26: A month later, I've had just one more spam comment. So it probably really was just coincidence.)

Abstract of "Advanced PostgreSQL, part 1"

2007-01-15T20:05:00.000-08:00

In December, Fujitsu made available a video of Gavin Sherry speaking on Advanced PostgreSQL. (Where's part 2, guys?) Here's some of the topics Gavin addresses, and the approximate point at which they can be found in the video.

[start]: wal_buffers: "at least 64"; when it's ok to turn fsync off [not very often]; how hard disk rpm limits write-based transaction rate, even with WAL
00:12:: wal_sync_method = fdatasync is worth checking out on Linux
00:13:: FSM [free space map], MVCC, and vacuum; how to determine appropriate FSM size; why this is important to avoid VACUUM FULL
00:22:: vaccum_cost_delay
00:26:: background writer
00:30:: history of buffer replacement strategies
00:37:: scenarios where bgwriter is not useful
00:41:: how random_page_cost affects planner's use of indexes
00:47:: effective_cache_size
00:49:: logging; how to configure syslog to not hose your performance
00:52:: linux file system configuration
00:58:: solaris fs config
1:02:: raid; reliability; sata/scsi; battery-backed cache ("for $100, you can triple the write throughput of your system")
1:08:: tablespaces
1:12:: increasing pgsql_tmp performance for queries that exceed work_mem and how to tell if this is worth worrying about
1:15:40: cpu considerations

Why SQLAlchemy impresses me

2007-01-12T22:04:00.000-08:00

One of the reasons ORM tools have a spotted reputation is that it's really, really easy to write a dumb ORM that works fine for simple queries but performs like molasses once you start throwing real data at it.

Let me give an example of a situation where, to my knowledge, only SQLAlchemy of the Python (or Ruby) ORMs is really able to handle things elegantly, without gross hacks like "piggy backing."

Often you'll see a one-to-many relationship where you're not always interested in all of the -many side. For instance, you might have a users table, each associated with many orders. In SA you'd first define the Table objects, then create a mapper that's responsible for doing The Right Thing when you write "user.orders."

(I'm skipping connecting to the database for the sake of brevity, but that's pretty simple. I'm also avoiding specifying columns for the Tables by assuming they're in the database already and telling SA to autoload them. Besides keeping this code shorter, that's the way I prefer to work in real projects.)

users = Table('users', metadata, autoload=True)
orders = Table('orders', metadata, autoload=True)

class User(object): pass
class Order(object): pass

mapper(User, users, 
       properties={
           'orders':relation(mapper(Order, orders), order_by=orders.c.id),
       })

That "properties" dict says that you want your User class to provide an "orders" attribute, mapped to the orders table. If you are using a sane database, SQLAlchemy will automatically use the foreign keys it finds in the relation; you don't need to explicitly specify that it needs to join on "orders.user_id = user.id."

We can thus write

for user in session.query(User).select():
    print user.orders

So far this is nothing special: most ORMs can do this much. Most can also specify whether to do eager loading for the orders -- where all the data is pulled out via joins in the first select() -- or lazy loading, where orders are loaded via a separate query each time the attribute is accessed. Either of these can be "the right way" for performance, depending on the use case.

The tricky part is, what if I want to generate a list of all users and the most recent order for each? The naive way is to write

class User:
    @property
    def max_order(self):
        return self.orders[-1]

for user in session.query(User).select():
    print user, user.max_order

This works, but it requires loading all the orders when we are really only interested in one. If we have a lot of orders, this can be painful.

One solution in SA is to create a new relation that knows how to load just the most recent order. Our new mapper will look like this:

mapper(User, users, 
       properties={
           'orders':relation(mapper(Order, orders), order_by=orders.c.id),
           'max_order':relation(mapper(Order, max_orders, non_primary=True), uselist=False, viewonly=True),
       })

("non_primary" means the second mapper does not define persistence for Orders; you can only have one primary mapper at a time. "viewonly" means you can't assign to this relation directly.)

Now we have to define "max_orders." To do this, we'll leverage SQLAlchemy's ability to map not just Tables, but any Selectable:

max_orders_by_user = select([func.max(orders.c.order_id).label('order_id')],
                            group_by=[orders.c.user_id]).alias('max_orders_by_user')
max_orders = orders.select(orders.c.order_id==max_orders_by_user.c.order_id).alias('max_orders')

"max_orders_by_user" is a subselect whose rows are the max order_id for each user_id. Then we use that to define max_orders as the entire order row joined to that subselect on user_id.

We could define this as eager-by-default in the mapper, but in this scenario we only want it eager on a per-query basis. That looks like this:

q = session.query(User).options(eagerload('max_order'))
for user in q.select():
    print user, user.max_order

For fun, here's the sql generated:

SELECT users.user_name AS users_user_name, users.user_id AS users_user_id,
    anon_760c.order_id AS anon_760c_order_id, anon_760c.user_id AS anon_760c_user_id,
    anon_760c.description AS anon_760c_description, 
    anon_760c.isopen AS anon_760c_isopen
FROM users LEFT OUTER JOIN (
    SELECT orders.order_id AS order_id, orders.user_id AS user_id, 
        orders.description AS description, orders.isopen AS isopen
    FROM orders, (
        SELECT max(orders.order_id) AS order_id
        FROM orders GROUP BY orders.user_id) AS max_orders_by_user
    WHERE orders.order_id = max_orders_by_user.order_id) AS anon_760c 
ON users.user_id = anon_760c.user_id 
ORDER BY users.oid, anon_760c.oid

In SQLAlchemy, easy things are easy; hard things take some effort up-front, but once you have your relations defined, it's almost magical how it pulls complex queries together for you.

.................

I'm giving a tutorial on Advanced Databases with SQLAlchemy at PyCon in February. Feel free to let me know if there is anything you'd like me to cover specifically.

MySQL backend performance

2007-01-12T13:49:00.000-08:00

Vadim Tkachenko posted an interesting benchmark of MyISAM vs InnoDB vs Falcon datatypes. (Falcon is the new backend that MySQL started developing after Oracle bought InnoDB.) For me the interesting part is not the part with the alpha code -- Falcon is competitive for some queries but gets absolutely crushed on others -- but how InnoDB is around 30% faster than MyISAM. And these are pure selects, supposedly where MyISAM is best.

Of course this is a small benchmark and YMMV, but this is encouraging to me because it suggests that if I ever have to use MySQL, I can use a backend with transactions, real foreign key support, etc., without sucking too badly performance-wise.

(It also suggests that people who responded to the post on postgresql crushing mysql in a different benchmark by saying, "well, if they wanted speed they should have used MyISAM," might want to reconsider their advice.)

Fun with three-valued logic

2007-01-10T12:43:00.000-08:00

I thought I was pretty used to SQL's three-valued logic by now, but this still caused me a minute of scratching my head:

# select count(*) from _t;
 count
-------
  1306
(1 row)

# select count(*) from _t2;
 count
-------
 19497
(1 row)

Both _t and _t2 are temporary tables of a single column I created with SELECT DISTINCT.

# select count(*) from _t where userhash in (select userhash from _t2);
 count
-------
   982
(1 row)

# select count(*) from _t where userhash not in (select userhash from _t2);
 count
-------
     0
(1 row)

Hmm, 982 + 0 != 1306...

Turns out there was a null in _t2; X in {set containing null} evaluates to null, not false, and negating null still gives null. (The rule of thumb is, any operation on null is still null.)

.................

I'm giving a tutorial on Advanced Databases with SQLAlchemy at PyCon in February. Feel free to let me know if there is anything you'd like me to cover specifically.

Good advice for Tortoise SVN users

2007-01-02T14:49:00.000-08:00

My thinkpad R52's screen died a couple days ago. I decided that this time I was going to be a man and install Linux on my new machine: all our servers run Debian, and "apt-get install" is just so convenient vs manual package installation on Windows. And it looks like qemu is a good enough "poor man's vmware" that I could still test stuff in IE when necessary.

Alas, it was not to be. My new laptop is an HP dv9005, and although ubuntu's livecd mode ran fine, when it actually installed itself to the HDD and loaded X it did strange and colorful things to the LCD. Things that didn't resemble an actual desktop. When I told it to start in recovery mode instead it didn't even finish booting.

That was all the time I had to screw around, so I reinstalled Windows to start getting work done again. Which brings me (finally!) to this advice on tortoisesvn: it really puts teh snappy back in the tortoise. Thanks annonymous progblogger!