Skip to main content

Materialized views in PostgreSQL

One thing I've wanted to write about for a while is materialized views in PostgreSQL. Materialized views are basically precomputed views; they're very very useful if you have an expensive query against data that doesn't change much. This is one of the tricks I use to keep the db overhead on Carnage Blender reasonable.

Oracle and DB2 are the only RDBMS products that can auto-materialize views for you, but you can rig them yourself in PostgreSQL or anything that gives you a powerful enough trigger + procedural language. (And if performance is crucial, you'll end up hand-optimizing them in Oracle/DB2 as well. At least, back when I used Oracle in the 8.1 days.)

But while looking for examples of a certain PL/PGSQL feature this morning, I found Jonathan Gardner's materialized views page (link may be down; see the archive.org copy) which covers the subject so well that now I don't have anything left to add. Nicely done!

Incidently, this is just one more reason real procedural languages in the database are critical for most substantial projects. Implementing materialized views in your application logic would be worse than a joke.

Comments

Ian Bicking said…
Would it be that hard to do in application logic? Depends somewhat on the database, I guess... but if you can intercept (via application logic) all the updates to that table (or otherwise efficiently detect that they have occurred) then you could manually create the "view" from the application side.
11:05 AM
Jonathan Ellis said…
First, it's on a par with the people who say "I maintain referential integrity in my application." Sooner or later you or your successor forgets and screws something up. It really isn't if, but when. I've seen this happen at three different companies, unfortunately.

Second, it's significantly less efficient to slurp data out of the db, perform computations, and stick it back in, than to do all that inside the database. Often you can wave efficiency under the carpet but presumably you wouldn't be doing this in the first place if you weren't optimizing a bottleneck.

Third, it's going to be very tricky to maintain transactional integrity while you perform the updates on the application side. At best you can get by with row locking but often you'll have to use the Big Hammer table lock. With PostgreSQL or Oracle-style MVCC everyone sees a consistent view of the data without locking.

Fourth, if you have multiple codebases accessing the data, see #1 and good luck.

Finally, maybe you are God's gift to programming and you really can do everything in app logic. And maybe your boss doesn't care about what happens when you leave and he has to hire a mere mortal to maintain it. But if you're happy using lousy tools, why are you using Python instead of C? :)
11:18 AM
Unknown said…
Materialized views sound a useful workaround, if your database doesn't have a query cache. If you do have a query cache, then you already effectively have eager or lazy materialized views (depending on your isolation level): Just use your normal view (or query) and let the database figure it out.

Snapshots are trivial to do in the application, and by definition are refreshed according to application logic (i.e. "only updated when refreshed", according to original article).
1:07 PM
Jonathan Ellis said…
MySQL's query cache works well, but it is a simple solution for simple problems.

For Carnage Blender, for instance, the biggest materialized view is stats derived from the main parties table and others. Dozens of queries hit this, joining to different other relations. Using the query cache approach would take far more memory (and memory is an issue for me: I already have a 5 GB database on a 4 GB motherboard) than letting PG cache the relations, with the expensive part precomputed in the mview, and derive what the queries need as necessary.
1:38 PM
Anonymous said…
I am merely an interested observer rather than an expert on this subject, but according to a guru of pgsql-general@postgresql.org (Tom Lane) there is no query cache in PG. I would be very interested if somebody could clarify this.
SWK
1:35 PM
Anonymous said…
One of the things missing in Postgres from Oracle is the ability to have the select rewritten on the fly to point to a MV table instead of the original table (half of the power of the MV).

I hope Postgres SELECT rule restrictions will be loosened in the future to allow for this type of redirection based upon noticing a SELECT is the same as the one that created the MV.
12:03 PM
Anonymous said…
The original post by Jonathan Gardner is gone. I have re-posted it at on my site Materialized Views in PostgreSQL
10:44 PM
Jonathan Ellis said…
Thanks for the pointer, although I don't think labeling someone else's work as Copyright 2008 yourself is kosher, even if his site is gone.
10:09 PM
Unknown said…
SQL Server supports materialized views, too. They're called "indexed views."
7:18 PM
Jake K said…
Good reaading
9:58 PM
Post a Comment

Popular posts from this blog

PyCon Python IDE review

I presented an IDE review at PyCon last Friday. It was basically a re-review of what I thought were the 3 most promising IDEs from the Utah Python User Group IDE review , to which I added SPE, which was by far the most popular of the ones we left out that time. The versions reviewed are: PyDev 1.0.2 SPE 0.8.2.a Komodo 3.5.2 Wing IDE 2.1 beta 1 I'd intended to base my presentation around a comparison of writing a smallish program in each of the IDEs, but the more I tried to make this not suck, the more I realized it was a losing proposition. Instead, I decided to try to focus on the features in each that most set them apart from the others (both positive and negative); this seemed more likely be useful. (I did a new feature matrix for this review, which is included after my comments. The slides I used are also up, at http://utahpython.org/jellis/pycon-ides.pdf , but aren't very useful absent video of the presentation itself. Hence this post.) PyDev PyDev has g...
43 comments
Read more

Why PHP sucks

(July 8 2005) Apparently I got linked by some PHP sites, and while there were a few well-reasoned comments here I mostly just got people who only knew PHP reacting like I told them their firstborn was ugly. These people tended to give variants on one or more themes: All environments have warts, so PHP is no worse than anything else in this respect I can work around PHP's problems, ergo they are not really problems You aren't experienced enough in PHP to judge it yet As to the first, it is true that PHP is not alone in having warts. However, the lack of qualitative difference does not mean that the quantitative difference is insignificant. Similarly, problems can be worked around, but languages/environments designed by people with more foresight and, to put it bluntly, clue, simply don't make the kind of really boneheaded architecture mistakes that you can't help but run into on a daily baisis in PHP. Finally, as I noted in my original introduction, with PHP, ...
30 comments
Read more

A review of 6 Python IDEs

(March 2006: you may also be interested the updated review I did for PyCon -- http://spyced.blogspot.com/2006/02/pycon-python-ide-review.html .) For September's meeting, the Utah Python User Group hosted an IDE shootout. 5 presenters reviewed 6 IDEs: PyDev 0.9.8.1 Eric3 3.7.1 Boa Constructor 0.4.4 BlackAdder 1.1 Komodo 3.1 Wing IDE 2.0.3 (The windows version was tested for all but Eric3, which was tested on Linux. Eric3 is based on Qt, which basically means you can't run it on Windows unless you've shelled out $$$ for a commerical Qt license, since there is no GPL version of Qt for Windows. Yes, there's Qt Free , but that's not exactly production-ready software.) Perhaps the most notable IDEs not included are SPE and DrPython. Alas, nobody had time to review these, but if you're looking for a free IDE perhaps you should include these in your search, because PyDev was the only one of the 3 free ones that we'd consider using. And if you aren...
36 comments
Read more