Sunday, April 06, 2008

My half-baked thoughts on Python web frameworks

I have been lucky to be able to fill our recent open positions with people who know Python as well as Java so now we are up to half the (6 person) company in that category and preferring Python, and 2 of the others have played with Python and liked it at least well enough to not object. So the boss has conceded that it makes sense to go the Python route for our next project.

We're going to be doing a web, "next gen" version of our existing client-server project, which is mostly simple CRUD but does have 1000+ tables in its current incarnation. So we really need something that can autogenerate 90+% of the CRUD or we will go insane.

The trouble is, I still don't really like any of the Python web options 100%. (I like the web options in other languages less, but I'm a perfectionist.)

Django is well documented, its admin app is something everyone else envies, and newforms looks decent, but the ORM blows and I'm not fond of the template engine either. (Pre-emptive pedantry: yes, I know I can "import sqlalchemy." Please stop saying that like it means something; I'm not interested in defining models twice -- once for real work with SA, and once for interop with the rest of django.) Apparently django-sqlalchemy got far enough in PyCon sprints that it's kinda usable so working on that would be an option. Of course even then there is no guarantee the django core would accept it into mainline, and maintaining it as a "vendor branch" would proably suck. If django used a dscm like Mercurial I might be willing to do that, but svn is just too painful so that is a real risk.

I don't see a way to generate a page containing just a CRUD interface for table X with the django admin app. The admin app really is a monolithic application, not something you can easily re-use pieces of.

Regexps suck for url mapping.

Pylons is not well documented and after keeping an eye on this for something like 18 months I don't think this is a problem that will be solved, for whatever reasons. On the other hand, SA + mako is a very sane default, and both of those are well documented so it's really only core Pylons that suffers from doc crapitude, and core Pylons is fairly small. IRC responsiveness mitigates this further.

Pylons still doesn't have a good CRUD (or even high-level manual form generation) solution, which has bugged me for even longer than the docs. I can't fathom how people can tolerate writing this kind of boilerplate in 2008. Formalchemy gets about 30% of the way there. DBMechanic requires TG2 atm, although apparently hacking it to run on Pylons may not be too much effort; I would guess around 20% of the effort to get the django-sa project really usable.

TG2 is of course very bleeding edge and although I like genshi's syntax in theory, in practice XML templates irritate the hell out of me. (Very verbose, xinclude sucks compared to "inheritance," and incorporating rich dynamic content -- i.e., user-generated, like forum posts, that needs to include html tags -- is a PITA. Not to mention that having to write "a > b" when you mean "a > b" bugs me all out of proportion to the actual inconvenience it inflicts on me.) Still, better than the django templates.

I'm skeptical that TG2 is a big enough value add to want to add it (in its unfinished state) as a dependency vs rolling our own on Pylons. But DBMechanic does look like it could be exactly what I want in a CRUD generator.

web.py seems like more of a tech demo than a real product. I don't see any signs of a CRUD or form generator. reddit, probably the largest web.py site at least in terms of page views, moved to Pylons.

Zope 3 is alone in being really production ready without running from svn. Grok does do a good job of smashing zcml and z3c.form looks okay but lives up to the Zope reputation of complexity. (Field managers, widget managers -- are these the same things? -- widget modes, ...) AFAIK relational dbs are still second-class citizens in zope, and with all due respect to zodb it is no postgresql. OTOH there is z3c.sqlalchemy which gives me hope. Finally: you have to manually restart zope (per the Grok tutorial) after changing your .py files? Seriously?

Bottom line, Zope might actually be a decent option if we had a Zope expert on staff but we do not and I am not willing to tackle the learning curve alone.

Nevow: form handling is in flux. The new hotness is "pollenation forms," but that is svn-only and the api "will probably change."

Zope and Nevow both have their own xml-based templates predecessing but similar to genshi. Something like Nevow's Stan is obviously useful for programmatic template generation but it's not yet clear if that's going to be something we need. Probably only if we have to write our own form generator. If so, I suspect ripping a standalone Stan out of Nevow would be straightforward.

(Spyce of course never really got any traction to speak of. It's time for me to let it go quietly into the night and leverage someone else's framework.)

Conclusion: I think porting DBMechanic to Pylons is our best option. DBMechanic seems designed to be more flexible than the django admin app. Django would be my second choice.

Corrections? Thoughts?

42 comments:

Doug Napoleone said...

I am confused about your comment on using CRUD in the django admin.

These are two very different things, and you should not try to implement anything on top of or in the django admin. Creating some generic CRUD views is not that difficult, but if you do not like the ORM or the template engine, and have no need for the admin (which I doubt you do), then django really is not a good fit as you have come to realize.

I would recommend just going with bare WSGI or Pylons and SQLAlchemy. It does not sound like you need any of the features that the current 'frameworks' provide.

Eric said...

I can only really respond to the Django stuff.

1. django-sqlalchemy not being included in core is not a problem since it's not a branch or a modification to Django itself. It's actually implemented using mostly the 3rd-party database backend library that is currently used to determine between sqlite, mysql, and postgresql. Using django-sqlalchemy gives you a fourth, sqlalchemy option, and that's where all the hooks into Django take place.

2. Why do you need Django itself to support your dscm of choice? Why not use one of the many excellent bridges that exist between svn and your dscm of choice? Especially since no real winner has been declared in this space, it's tough to see this as a real problem.

3. For the autogeneration stuff, I think you're looking for a combination of "django-admin.py inspectdb" (which generates a models.py file from tables in the database) and generic views. There are generic views for creation, updating, and deleting, as well as a host of reading operations--list views and date-based views. Since templates are always going to be slightly different for each user, that's really going to be the only thing you spend much time on, if you use generic views correctly.

4. The newforms-admin branch is making great steps in transforming the admin into something that is less monolithic. It's fairly stable and I'd highly recommend checking it out.

5. Why don't you like using regexes for url patterns? Trying to match a{n}b{n}? :) Honestly though, why don't you like it?

By the way, thought your talk on PyTriton was fascinating. Kudos on all that hard work!

Andrew Ingram said...

I'm also interested in why you don't like RegExps for URLs.

A while ago when I was looking into crafting well-designed URLs I was trying to come up with a simple notation for defining a URL schema. Then I realised it already existed in the form of Regular Expressions.

Sean said...

Couple of things:

1) What is the big win for you of SQLAlchemy over Django ORM?Aggregation and other more complex queries? Or is there something more deepseated you don't like about the Django ORM here?

2) Although I agree that saying 'just import sqlalchemy' doesn't cut it, the same is not true for alternate templating engines. It's dead easy to use Django with Genshi/Jinja/others and unlike with the ORM you don't really lose any cohesion by doing so.

Martijn Faassen said...

Thanks for the note on Grok - it's always interesting to get comments! I'll create a launchpad issue on automatic restarts - I didn't realize that this is such an important issue for some. Done.

A few notes about Grok: Grok ships with integration of zope.formlib (not z3c.form), and less intimidating than z3c.form. z3c.form is indeed classic Zope 3 "power over simplicity", though I think we'll be able to tame it someday with Grok. Our formlib integration isn't as well documented as we'd like, but documentation can be found here.

On SQLAlchemy integration: I've recently done a project with Grok and SQLAlchemy and I'm quite pleased with how easy the two integrate (ZODB and PostgreSQL in the same application, sharing the same transaction!). SQLAlchemy plays very well with Grok. Hopefully we'll be able to start sharing a good default story on this soon. It's really of importance to the Grok project to have this work well out of the box for people.

Fabian Neumann said...

Count this as another one who want's to know the bad of regexps for URL matching. I think it's the ideal way.

Joe said...

A great starting point for the z3c stuff is http://docs.carduner.net/z3c-tutorial/. The other useful useful knowledge for zope is adapters and the component architecture.

Phillip von Weitershausen's book "Web component development with Zope 3" is also great.

voltron said...

Hi! Have you tried web2py? It addresses a few of the things you mentioned.

http://mdp.cti.depaul.edu/

Jonathan Ellis said...

Thanks for the comments, everyone.

Doug: well, the django admin is all about automating a certain use case of CRUD, right? So I think a certain amount of confusion is natural for a first-timer. But yes, getting this kind of confusion cleared up by people who _are_ experts where I am not is the main reason I posted this. :)

Eric: thanks for the clarification about django-sa, and good point about the dscm/svn bridges, although I think only git's is really first-class, and I don't care for some other things about git. I'm not just picky about web frameworks! :)

It does look like generic views + formpreview does a substantial amount of what I want. Too bad inspectdb is static rather than dynamic.

Sean: SA can handle all kinds of mapping and queries that django cannot. I do have production experience with Rails + ActiveRecord and I ran into "dammit, AR can't do this either?" frustrations literally daily. I honestly have no idea how people can be satisfied with such limited solutions. I imagine that your brain eventually starts to think in terms of what your ORM can do instead of what the database can do, but why cripple your thinking when a best-in-class ORM does exist? The Django ORM and AR are roughly at the same level with AR probably having a slight edge. See my "why SA impresses me" post for one example (although there are easier ways to accomplish this in modern SA). Additionally, the unit of work pattern is much better than manually saving every updated object.

Yes, the templates thing bugs me a whole lot less than the ORM one since as you say there really is no obstacle to just using something else. But as Doug says why bother with django at that point? :)

Martijn: thanks for the good news about SA + grok.

All: I've written a lot of regular expressions and while I like to think I'm pretty good at doing so, they've always stayed slightly "write-only" for me. The thought of poring over dozens of re's in a big app trying to figure out exactly why my url isn't being mapped where I expected does not appeal to me. I find Routes more readable.

Jonathan Ellis said...

I looked briefly at web2py but only briefly. There is really no excuse for writing a home-grown ORM in 2008. At least when django started SA did not exist yet; the only thing you can fault django for is stubbornly refusing to switch to something an order of magnitude better. :) The argument for using Pylons as a base instead of rolling your own MVC is almost as strong, so I have to wonder how good it can be when Massimo wasted so much time re-inventing the basics.

Deuce868 said...

>>> 1) What is the big win for you of SQLAlchemy over Django ORM?Aggregation and other more complex queries? Or is there something more deepseated you don't like about the Django ORM here?

I've just started messing with Django and right off the bat, the single key pk thing bit me.

Anonymous said...

Stop wasting time looking for silver bullet frameworks. You're always going to be disappointed and end up reading more code then you need. I've built what I need on top of Cherrypy which is very well documented, handles the tricky server and threading parts, is very extensible and gets out of my way.

Jonathan Ellis said...

(Michael Trier just posted a django-sa update.)

Jonathan Ellis said...

Past a certain point I agree that looking for a silver bullet is a waste of time, but it's definitely worth it to spend a couple days looking for the best fit before committing several man-years to something.

Anonymous said...

Another option could be werkzeug, utilizing sqlalchemy and wtforms.

Take a look at their routing system as well. (I couldn't post an example of an url_map here because angle brackets aren't allowed.)

Oh, and of course, as the templating language I use jinja which was also written by the authors of werkzeug. Jinja is an enhanced version of django-templates.

Mark Ramm said...

Hey Jonathan,

I note that you say that DBMechanic is TG2 specific -- I don't think that's the case really. Chris has reported success running it with Pylons and Grok, and the goal is to support all the major frameworks.

I think you personally at this moment may be best off with Pylons and not TG2, but I also think it's important to mention that TG2 provides things like DBSprockets integration, automatic form handling, and the like out of the box.

We can do those things because we have a known core of standard components, while Pylons gives you the freedom to choose your own.

We've got more plans along those lines, from user registration, to other larger site-components. TG2 will provide a lot of things that Pylons just can't.

With that said, we fully intend to support two options in one place. We've decided to anoint Mako as our "second" templating language -- because sometimes you just need the performance.

But, Pylons will offer something that TG2 can't -- the abilty to swap out components. I mean you could swap out the ORM in TG2, but you'll loose all the integration featurs that we've added on top of Pylons.

Sean said...

I honestly have no idea how people can be satisfied with such limited solutions. I imagine that your brain eventually starts to think in terms of what your ORM can do instead of what the database can do, but why cripple your thinking when a best-in-class ORM does exist?

The way I work is using Django ORM for the obvious stuff: queries to retrieve sets of objects; and then stored procs and/or hand-coded SQL for anything more complex. I've never worked with an ORM that made sense to me for anything beyond basic querying, but then I've never tried SQLAlchemy. You've convinced me I should give it a good look, especially as the django-sa bridge seems to be coming along nicely, so using it with my existing Django projects may soon be fairly straightforward.

Anonymous said...

easy solution:

switch to rails or merb

Matt said...

Poor Spyce. I had high hopes for it. It's one of the reasons I started learning Python (in order to get away from PHP).

mdipierro said...

It is not correct to say that web2py reinvented the wheel. It is more like web2py wanted tires hence could not use existing wood wheels but needed its own iron core wheels.

web2py is the only framework that participated to the www.flourishconf.com rumble (develop a web app in 24 hours) and won because all the other frameworks chickened out. This https://mdp.cti.depaul.edu/SurveyRumble was developed in web2py in 24hrs (actually 12hrs because I slept and ate too meanwhile).

SQLAlchemy is a great ORM but was not appropriate for web2py since its components required a tight integration that could not have been achieved without modifying SQLAlchemy. Anyway, it is not like it is difficult to write an ORM or learning to use one so it was not much time I wasted. btw, the web2py ORM does migrations, supports Oracle (including limitby), does left joins, and its is much much much easier to use than SQLAlchemy.

web2py is based on wsgi and until version 1.18 it was based on the httpserver from Paste. Turned out that the cherrypy 3.0 wsgiserver was faster so I switched to that. All my users supported this switch and noticed an improvement.

It seems pointless to me to criticize web2py because "you do not like it" and because "it did not use your favourite project". Tell me instead which functionality (you think) is missing in web2py or which benchmark you performed and have found web2py slower than another framework, if you can.

Jon said...

There has been a good deal of discussion in the Pylons and surrounding communities about form generation. Generally it comes down to "We can have something that works for simple cases, but any time you need anything remotely complex you're gonna want to drop down to HTML anyway, so why bother?" Same reason we've never bothered to implement Rails-style scaffolding.

I am happy to report my ModelWebHelpers idea is being included in the next release. That is, you can use a special variant of the input tag webhelpers that will automatically fill in values from the model record, if it's present. Check out http://paste.lisp.org/display/58562 for a sample of something close to how it's going to work. (We're using the 'special object' approach variant at the bottom.)

Jon said...

mdipierro, is it really fair to say that "all the other frameworks chickened out"? Or is it perhaps more accurate to say that "all the other frameworks had never heard of it"? I know I haven't, and I like to think I'm pretty active in the Pylons community. Looks like you had a Rails contestant (pretty dang easy) and a CakePHP contestant; if this is truly a well-known contest, where's the Lisp guys, or the Merb guys?

schmichael said...

What about CherryPy? Its not a monolithic framework so you can easily use any ORM (or none at all), as well as any templating language.

The default dispatcher is Object/Method based, and I quite enjoy it. It has a Routes based dispatcher which is quite popular as well.

I need to get these uploaded to http://cherrypy.org, but here are a bunch of case studies for CherryPy:

http://groups.google.com/group/cherrypy-users/browse_thread/thread/1774049cc582a595

Its an excellent choice for people whose needs don't fit the cookie-cutter megaframeworks.

(Full disclosure: I also use and quite enjoy Django. "Right tool for the job" and all that. ;) )

jerf said...

Regardless of your eventual choice, remember that any configuration that exists as a programmatic specification can be created on-the-fly from an existing source.

Using Django as an example, nothing stops you from building Models from a DB specification; it'll take one person who understands a bit more about Python than most people, one person who understands how metaclasses work, but if you autogenerate classes from the DB, it'll work.

(The default ORM will probably still not be sufficient for your needs; I'm just using an example.)

Jonathan Ellis said...

> it is not like it is difficult to write an ORM

Thanks, now I know that staying away from web2py was a good decision. :)

(Sure, it's easy to write a simplistic one that doesn't let you take advantage of what relational databases are good at. But I'm not interested in using those. Again, see Why SQLAlchemy Impresses Me for one example.)

Chris Perkins said...

DBSprockets is supposed to answer a lot of the problems that people have had with moving from one framework to another. It offers capabilities that many other form-creation libraries do not. Namely, it can handle validation, Automated foreign key and many-to-many table drop-down creation. It has a simple primitives interface which gets you up and running quickly. It can also create tables with pagination automatically, including the foreign key/many-to-many relationships. All this in a portable package that allows you to run with any orm, on any wsgi stack.

As a matter of fact, the start of the port for DBMechanic to pylons has already been done. We got this up and running at Pycon last month. Here is a how-to: http://code.google.com/p/dbsprockets/wiki/DBMechanic (look at the bottom for pylons)
No, it's not polished, but it does work.

TG2 offers automatic crud based on dbsprockets. The general template will generate a listing page, and edit page, and a create page. This is a pretty useful tool for stubbing out a website, and something that you don't get with pylons.

TG2 does support Mako, by the way, and it is not hard to integrate. Personally, I think you get a lot of bang for your buck with TG2, and I wouldn't shy away from it based on a notion of flexibility, or value added.

About Django-admin. A few weeks ago we got a proposal for GSoC which basically took the Django Admin syntax and laid it on top of dbsprockets. After looking at it, I realized I could probably implement it in about a weekend. However, I thought it violated the DRY paradigm, so I refrained. Basically, I am saying we can make a pretty simple django-admin type interaface for TG2, but it doesn't *feel* right at this time. Maybe someone will change my mind.

Lastly, we are working on the docs. I have been working with one of our GSoC perspectives on fleshing out the ToscaWidgets documentation, adding ajax widgets, and creating a bunch of how-tos to help get people up and running.
http://docs.turbogears.org/2.0/RoughDocs/ToscaWidgets

Others have been working very hard on the TG2 docs as well. We realize that docs are a problem, most people in the wsgi world don't have a large publishing company behind them, we are mostly engineering-types who just want things to work, and provide our users with well-tested releases (as opposed to living with hot_action). Our wiki pages are all freely editable, and I would hope that developers using TG would contribute more in the future.

In short, dbsprockets supports a lot more frameworks than you might think, (even django is on my list to get working) and TG2 offers many more integrated tools out of the box than you might think. Feel free to email me and ask me about either.

cheers.
-chris

Jonathan Ellis said...

This thread is what gave me the impression that DBMechanic support for Pylons wasn't complete yet. Thanks for clearing that up, Chris.

Anonymous said...

2Jonathan Ellis
we2py's orm not bad at all.
why are you blame it? only beacause it isn't sqlalchemy? :)

Anonymous said...

"All: I've written a lot of regular expressions and while I like to think I'm pretty good at doing so, they've always stayed slightly "write-only" for me. The thought of poring over dozens of re's in a big app trying to figure out exactly why my url isn't being mapped where I expected does not appeal to me. I find Routes more readable."

This is really not a deal unless you plan on writing some hideous urls. Especially if you plan on using a simple rest style, your urls.py classes will look succinct and easy to understand.

I was a little worried about the regex thing too, since I don't care for them, but it has not been an issue at all, and it's damn handy to have them there when you need them. I wouldn't trade them at all now that I have used them.

Jon said...

Seems to me that a lot of the people here praising the power of regexps for urls probably haven't looked at Routes. Routes uses regexps underneath, so it has all the same power, only a more readable syntax and much more flexibility.

Derek said...

so your main problem with Zope 3 is that you have to learn how to use it? Get real - you'd have to learn how to use every framework. You do not have to restart zope after editing python files, I don't know where you got that from.

Graham Higgins said...

++ Pylons is not well documented and after keeping an eye on this for something like 18 months I don't think this is a problem that will be solved, for whatever reasons.

In a previous post, you wrote:

++ Poor documentation of core Pylons ... I had to use the source several times. ... The first tutorial overcomplicated things, showing how to configure things to handle semi-obscure requirements, without explaining those requirements or simpler alternatives.

Is this still the core of your objection to Pylons docs? If you have additional observations to make, I would be very interested to them (the email address below is valid for a week).

Cheers,

Graham Higgins

gjh-dated-1208191473.090b82@bel-epa.com

Rok Garbas said...

i'll go zope/grok if i were you. take closer look in ZCA - zope component arhitecture. its a powerfull tool to build also not zope applications.

and maybe a link of zodb vs postgres benchmark ... http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks-revisited

enjoy ...

Jonathan Ellis said...

The core of my objection to pylons' docs is that there's a wiki page here, a wiki page there, but it doesn't look like anyone has stepped in to make things feel like a unified whole. This may be inherent to the wiki format.

cavanaug said...

Mercurial distributed mirror of django available at http://hg.dpaste.com/

Martijn Faassen said...

Wow, quite a discussion here! On regexes versus routes: Zope takes another approach based on traversing objects. With Grok we've made it quite simple.

You can define a 'traverse' method on content objects (or if you want, against existing objects defined in completely other packages thanks to the power of the component architecture), which takes a name (which is a URL step), and returns either an object to traverse into, or None (traverser doesn't know what to do, fall back on whatever is there for this object; it could for instance be a container or have views).

It also turns out to be quite easy to use things like zope.location.location.locate() to give database-driven objects a 'location' in URL space, meaning you can very easily generate URLs for these objects.

The nice thing about the traversing approach is that (infinitely) nested hierarchies are very easy to create. You can also easily move objects around in model-space without having to remap a lot of URLs, and combine objects from different projects into a single application.

A drawback is that it's a harder to control what URLs get published (though the skin concept helps quite a bit). The nice thing about a routing approach is that you get a limited amount of places to look at to see what URLs the application supports.

My apologies if I said the wrong things about routing though; I'm quite sure there are solutions to all the issues I listed. I am making the case that traversing makes these cases feel very natural.

Martijn Faassen said...

Oh, by the way, concerning Zope using ZPT: that's the default for Grok too, but Grok has featured Genshi integration (by an extension) for quite a while now. :)

ionelmc said...

Also, there's another problem you haven't covered at all in your post:
Content steaming.

Sadly, django doesn't support any form of it.
Pylons has some issues but it's getting there.
The WSGI spec supports streaming, unfortunately a lot of middleware isn't compliant.

Krys said...

Hey Jonathan,

I'm not sure if this meets your needs, but your post reminded me of the tgcrud addon for TurboGears (1.0). I think these are docs for it http://docs.turbogears.org/1.0/CRUDTemplate.

TG1 may not be the new hotness, but it is stable and tgcrud does seem to support SQLAlchemy.

Anyway, I have not used tgcrud myself, but since I did not see anyone else mention it, I thought I would.

HTH,
Krys

Anonymous said...

Thanks for letting us in on your thinking.

We're making the same decision. Our plan is to do some prototypes with TG1 right now but build our real system in TG2. A little bleeding-edge, but all the important components look mature, so it's really Pylons + SA + Elixir + Genshi with some glue. And, if the glue ends up being messier than our current impression, we can fix it and contribute back to the project.

The only questionable side is ToscaWidgets which seems poorly-documented. We'll evaluate that a bit later. If so, we'll have to wrap some parts of Dojo ourselves.

Frank said...

Hi Jonathan, Thanks for sharing your thoughts on this. I am having a similar problem. I have been using django, and like that I can get the admin interface going quickly. However, I am considering a project that I am not sure if it will go the direction of a web front-end or a desktop app. I would like to use an ORM, but I don't think django's orm can be ported to a desktop app. Plus, as you say, it's not the ideal ORM. I would love to find a nice ORM (python), and then slap on a web framework that generates the crud screens. Then if the web front-end works well, then I can run with that. Otherwise, scrap it.

Any advise?

Frank

Jonathan Ellis said...

Frank, check out FormAlchemy. It sounds like it's a good fit for what you want (rendering CRUD forms from SQLAlchemy models):

http://spyced.blogspot.com/2008/10/formalchemy-10.html

http://spyced.blogspot.com/2008/10/small-admin-app-for-pylons.html

The formalchemy mailing list is a good place for followup questions.