I spent last week with my wife and her family in Poland (my wife is Polish). Her parents do not speak English, or any other language than Polish so communication is problematic beyond the thank you, you’re welcome, yes and no thank yous. My wife also spends a lot of time meeting with her friends, so I typically spend quite some time staring at the wall among people who are speaking a language I do not know well enough yet.
So, recently I’ve been coming up with little one-week programming projects for the weeks we spend there. These projects do not have to lead to anything in particular, but give me a good amount of time to take a deep dive into something I do not ordinarily have time for.
Last week my project was building a web application and framework with Clojure and MongoDB. I already had a plan for a web application in mind before, and already read up on Clojure (through the excellent Programming Clojure book) and played with it a little bit. I have also been interested in non-relational databases for quite some time and before have played with Google AppEngine’s DataStore and CouchDB. Similar to CouchDB, MongoDB is a document-oriented database, but it has more "conventional" querying methods than CouchDB, still not SQL though.
For a week on-and-off I went to work.
There were basically two components to this project, first getting to know MongoDB and second learning to build a nice internal DSL in a functional language, specifically, a lisp (I never seriously learned a Lisp before, only supervised practical sessions for a class using Scheme). As a case study I came up with a nice not-too-complicated web application to build. I won’t go into that application in this post, it’s still secret (wooh!).
Alright. MongoDB is written in C++ and has readily compiled binaries for most platforms available and very easy to install (simply extract and run). It is known for its good performance and used by many companies, including Sourceforge and Disqus (the comment system I use on this website). After starting the server, the easiest way to start interacting with the system is the mongo javascript console.
Let me demonstrate it by simply showing you a sequence of commands (prefixed with >) and outputs:
> use people
switched to db people
> db.Person.save({name: "Zef Hemel", age: 26})
> db.Person.save({name: "Justyna Hemel", age: 26})
> db.Person.find()
{"_id" : ObjectId( "4b3b51c24905573d69b9bd67") , "name" : "Zef Hemel" , "age" : 26}
{"_id" : ObjectId( "4b3b51d64905573d69b9bd68") , "name" : "Justyna Hemel" , "age" : 26}
Note that the people database did not exist yet, and was in fact created when it was first used, similarly, the Person collection ("mongoose" ;-) for table) was automatically created when I saved a first record to it. Like other document databases, collections are schema-less. Now that we have some data, we can start querying:
> db.Person.find({name: "Zef Hemel"})
{"_id" : ObjectId( "4b3b51c24905573d69b9bd67") , "name" : "Zef Hemel" , "age" : 26}
So querying happens by passing the find function a map of keys and values that must match in a document. This notation gets slightly weird when looking for age ranges, for instance between 20 and 30:
> db.Person.find({age: {$gt: 20, $lt: 30}})
{"_id" : ObjectId( "4b3b51c24905573d69b9bd67") , "name" : "Zef Hemel" , "age" : 26}
{"_id" : ObjectId( "4b3b51d64905573d69b9bd68") , "name" : "Justyna Hemel" , "age" : 26}
So there, as value of the property age, we give it another map with special operators $gt and $lt, which stand for… greater than and less than! It’s a bit odd, but it’s easy to get used to (and screaming to be wrapped in some nicer syntax on a language level).
Because no indexes have been defined on the collection yet, this lookup is still rather slow. However, indexes can easily be defined:
> db.Person.ensureIndex({name: 1})
true
This defines an index on the name property in ascending order (-1 would be descending). The index order only matters when putting indexes on multiple columns and sorting on some of them, or so the manual tells me.
So, MongoDB is fairly straight forward to play with, easily create new collections, add properties and so on. Intuitively it feels like a good match to a dynamic language, such as Clojure.
Clojure is dynamic functional language for the JVM. As mentioned, it is a Lisp. It comes with a nice interactive REPL to experiment with. The most interesting thing about Clojure from my point of view, as somebody doing research into domain-specific languages, is the ability to create domain-specific languages with it. As you will know, the syntax of Lisp is extremely simple and mostly defined by its functions and macros.
There is already a simple Clojure web framework called Compojure, which is basic but quite powerful. For my application I decided to build some layers on top of Compojure. First of all, compojure only deals with the web side of things and not with database stuff. For MongoDB there is CongoMongo, a simple Clojure interface to MongoDB. This turned out the be far from complete, however, so I branched it and added a bunch of functions to it.
I decided to call my little framework Adia, it’s available for download from github, see the Readme there for installation instructions, there’s no documentation yet, there is however a simple wiki application in the examples directory.
Although MongoDB does not enforce any schema, it seemed like a useful thing to define a simple entity language anyway, if not for the database itself, for me, as documentation and possibly for automatic form generation and data validation, later. This is what it looks like:
(defent Page
[:title :string {:unique true}]
[:author :string]
[:text :text])
As can be guessed, this defines a Page entity with three properties: title, author and text.
As you will be aware, I’m a developer of WebDSL, a DSL for building web applications, and came to appreciate its simple page and template abstractions. Although implementing actions in a WebDSL fashion would be against the functional character of the language, I did add a page abstraction, except I call them webfns, defined with defwebfn (similar to defn, to define a Clojure function):
(defwebfn say-hello [nam str]
(str "Hello, " nam))
This defines a web function with one parameter: nam, which is coerced to a string value through the str function. Similarly, every entity definition also defines a function with the same name that can coerce the URL representation (identifier) and retrieve its value from the database, e.g.:
(Page "31108a33ee093a4bdd7b5900")
Retrieves the page object with ID "31108a33ee093a4bdd7b5900". This can be taken advantage of in web functions as follows:
(defwebfn show-title [p Page]
(str "Title: " (:title p)))
These webfns are available through a URI based on their name and the last part of the namespace they were defined in. For instance, when a webfn show is defined in namespace myapp.user, it will be available through "/user/show". Any namespace ending with .index, or webfn named index, are bound to the root, e.g. webfn index in myapp.user results in "/user" and webfn index in myapp.index is bound to "/".
Templates are, of course, simply functions with parameters. Compojure comes with a rather nice alternative HTML representation using Clojure vectors:
[:a {:href "/"} "Link text"]
Which can be used to define a main template:
(defn main-layout [title & body]
(html
(doctype :html4)
[:html
[:head
[:title title]]
[:body
[:h1 "Header"]
[:hr]
body
[:hr]
"© Zef Hemel"]]))
Of course, vectors like these can easily be combined with regular function calls, to build pages. Here is an example of an index page with title "Wiki home", displaying a list of current pages and a form to add a new one:
(defwebfn index []
(main-layout
"Wiki home"
[:h1 "All wiki pages"]
[:ul
(for [p (query model/Page)]
[:li (navigate [show p] (:title p))])]
(form [handle-add]
[:h1 "Add a page"]
[:div "Title: " (input-string :title)]
[:div (input-text :text)]
(submit-button "Add page"))))
This will be rendered roughly as follows:
The actual adding happens in the handle-add function:
(defwebfn handle-add []
(let [p (databind
(model/Page
:author (get-session :username))
*form* [:title :text])]
(redirect [show (persist! p)])))
Additionally, I can define an access control rule for handle-add:
(defac handle-add (get-session :username))
Which says that only if the session key :username has a value, i.e. the user is logged in, a page can be added, resulting in the handle-add only to be available to logged in users. In addition, the form on the index page will be hidden when the user is not logged in. Similar to WebDSL’s navigates.
Lessons learned
MongoDB is a nice and simple NoSQL database system and when you’re in the right document-vs-row no-join-required mindset, it’s easy to work with. I found that it also works well with Clojure, initially I played with Clojure and MySQL a bit, which also works fine. Still, I found it a bit slower to iterate because you keep creating and dropping tables and adding, modifying and removing columns in your table as you’re developing the application. Database migration is a pain. In MongoDB this is less of a problem I have found.
Clojure is quite a nice, elegant, simple language and macros are a very powerful way of defining new "syntax" for your own little domain-specific languages. The syntax of the language is the user interface to the developer and is therefore important. Not everybody is a fan of the Lisp syntax and it definitely takes some getting used to. Lisp programmers say you should see through the parentheses and instead look at indentation to extract meaning from programs. That works, but bites you in the ass when you do some s-expression manipulation and misplace some parenthesis and do not let your editor (I used vim with vimclojure) re-indent your code. This happened to me a few times and on occasion took me quite some time to debug.
The Clojure syntax is concise, maybe too concise. Sometimes I find it hard to e.g. see what piece of code is part of the true and which is of the false branch of an if-statement, an else keyword can be useful to make code easier to read.
Homoiconicity is cool. The defwebfn macro, in addition to defining a function and doing some other stuff, also keeps the original list structure that defines the web function in memory (the source code, as it were). The access control module takes advantage of this by taking this code, wrapping an if statement around it and recompiling it at runtime. Model transformations at runtime! Potentially more advanced program transformations can happen in this way.
Are Clojure DSLs as flexible as external DSLs? Not really. First off, you’re locked into the Lisp syntax. Second, checking is rather limited and error messages not always extremely helpful, similar to DSLs in other languages, such as Ruby. Clojure does do compile-time symbol lookups which is helpful, but beyond that a lot of errors are detected at runtime. Macros are evaluated at compile time and can therefore check a few things then, but this checking is limited to the "AST" representation of its arguments. Clojure is a dynamically typed language, so checking in general is problematic. And third, you’re bound to the JVM (or with more effort CLR), you can’t target multiple platforms.
Meta-programming in Clojure is cleaner than in many other languages such as Ruby and Python. Most of it happens using macros which are fairly clean, if used well.
As I suggested before, libraries and internal DSLs like these are great ways of prototyping abstractions. They’re easy and quick to implement. Access control was added in about 20-30 lines of extra code, OpenID authentication took about 30 (through use of JOpenID). This makes Clojure a great language to try out and play with abstractions. I find that in Stratego, which we use to implement WebDSL, this is still problematic due to the fact that (1) it is a separate language, so you have to make mental jumps between Stratego and the target language, e.g. Java, and (2) long compilation times of the ever growing WebDSL compiler.
As mentioned, if you’re interested in Adia, you can download it and play with it yourself. Documentation is essentially non-existent as of yet, but the wiki example demonstrates its basic features.
