Common Lisp web development and the road to a middleware
I’ve been looking recently for a Common Lisp system, which would allow me to expose a service I am developing over a REST API and honestly the options available right now in the Common Lisp ecosystem may be a bit overwhelming at first.
If you are like me and going through the same exercise of finding a
system to develop a simple REST API you will be hearing a lot about
The list goes on and on … Prepare yourself for some reading what exactly each system is and what it is meant to be used for. Hint: some of these are web servers, others are web frameworks, and others are micro web frameworks.
The Common Lisp Cookbook does provide a good introduction to the various systems and what they do, so make sure to check it out.
The State of Common Lisp, 2022 also provides a good overview of the different options at your disposable, but after reading it you will probably be even more confused as to which one to use.
The Why Hunchentoot instead of Clack? article also makes some good points to think about when considering one or the other, so check that one as well.
In my case I was looking for a simple web framework, which would allow me to develop a simple REST API. Something along the lines of ring in Clojure, which I’ve used before, when I was doing some Clojure.
With that said I’ve looked into
caveman first, as this seemed
to be what people have been recommending.
caveman is a
“batteries-included” web framework for Common Lisp. It appears to be
the Django for Common Lisp, and
comes with nice goodies such as template systems, database
integration, configuration system, project skeletons, routing macros,
etc. I’ve tried out
caveman and while I do liked it, it is a bit too
much for my simple use cases.
Another thing to keep in mind about
caveman is that it is based on
clack, which is based on
lack. What that means is that now you
have to learn all the abstractions the framework is built on top of.
Again, comparing that to ring
it is a bit of an overkill to me, so I’ve decided to keep looking
elsewhere for now.
One thing worth mentioning here is that if you decide to use
as a web framework it abstracts away the web server specifics by
providing a unified API. What that means is that you could be running
hunchentoot or other servers from your
clack app, which is a nice
Both of them provide interesting features, and approach the same
problems in different ways. For example in
snooze a route is modeled
around CLOS and uses generic functions and the accompanying methods,
which is really nice, because you could decorate your HTTP routes with
:around methods. I really like that one! I’m
planning to spend more time soon with
snooze, but on the surface -
it seems to be what I look for.
ningle is nice as well, and one of the features that are interesting
to me is the ability to define additional requirements per
ningle is easy when following the README documentation,
but you will need to read more about
lack as well in order to fully
understand the design choices in
ningle. An honestly
provide much of a documentation, so this makes it a bit of a challenge
initially. Compare that with the ring
documentation, which is
great in my opinion, as it provides everything you need to get
Okay, this opening paragraph ended up much longer than what I’ve planned initially, but I wanted to give you a bit more context as to what I was going for.
In a future post I may write more about my experience with web development in Common Lisp, but before that let’s get to the point of this post – how to create custom middlewares in ningle, as this has been something I really wanted to get working, and the system doesn’t support it.
First, what is a middleware? A middleware is a function which wraps an
existing application and returns a new application. A typical use case
for a middleware is to provide logging capabilities for your web
service, where each request/response is logged as it passes through
between the client and the server. Other use cases for middlewares is
the ability to push (or inject) additional context to your HTTP route
handlers, e.g. a database connection which will be re-used by all HTTP
handlers (routes in
The documentation of ningle, clearly states that “you can use other Lack middlewares with ningle”.
While that is true, and you can use middlewares in
ningle, and these
middlewares can interact with the surrounding HTTP request
environment/context before they hit the actual HTTP handler/route, it
appears that the HTTP handlers/routes in
ningle are somewhat limited
when comparing them with a handler/route written in
Here’s an example of a HTTP route/handler from the lack documentation:
(lambda (env) (declare (ignore env)) '(200 (:content-type "text/plain") ("Hello, World")))
It is just a regular lambda function, which takes a single argument –
the environment of your
app. Now, you can
use this environment, and in fact that’s what middlewares do in order
to push additional context to your app. This is what a typical
middleware looks like in
(defun middleware (app) (lambda (env) ;; preprocessing (let ((res (funcall app env))) ;; postprocessing res)))
You can then use this middleware to wrap an existing app and provide a different behaviour when your route is called.
You could create a middleware that looks like this.
(defun middleware-message (app) (lambda (env) (setf (getf env :message) "A message from the environment") (funcall app env)))
If you enable this middleware using
lack:builder now you can access
the message that was pushed into the environment by our middleware.
(lack:builder #'middleware-message (lambda (env) `(200 () (getf env :message))))
While this is a pretty contrived example, I hope you get the idea. You could use middlewares for things such as pushing a database connection, so that your HTTP routes can use it, when needed.
So, back to
ningle the HTTP handler/route looks like
(defvar *app* (make-instance 'ningle:app)) (setf (ningle:route *app* "/hello/:name") #'(lambda (params) (format nil "Hello, ~A" (cdr (assoc :name params)))))
Again we see the well-known lambda handler, but this time the
lambda-list is different – your
ningle route will receive the HTTP
request params, and not the surrounding environment. And that is a bit
of an issue here, because if you want to create a custom middleware,
which touches the environment there is no way your route will ever be
able to get to it.
The README provides a section about ningle context, which seems to be what you should use, but after trying that out that doesn’t seem to be the case. Here’s the example from the README.
(setf (context :database) (dbi:connect :mysql :database-name "test-db" :username "nobody" :password "nobody")) (context :database) ;;=> #<DBD.MYSQL:<DBD-MYSQL-CONNECTION> #x3020013D1C6D>
You may end up with the impression that this thing will set things up
for you and you can use the
context from within the routes and you
will be right. The problem is that
(setf (ningle:context ...)) works
only within the routes. And that is too late in the game already.
The funny thing is that the
(ningle:context) is derived from the
environment that we’ve seen before in the
lack routes, but it
is stripped down to just a few things – the
session, which is set up by the
So, to sum it up – if you need to create custom middlewares in
ningle, which interact with the
environment you are stuck, because
your HTTP routes won’t be able to use it. This appears to be a known
thing and was reported ~ 7 years
ago. Before I
ningle of the list of frameworks I’d like to use
I’ve decided to give it a stab and see if I can make it work.
Checking the code, the method of interest is this
which defines a method for
lack:call of our ningle app.
(defmethod call ((this app) env) "Overriding method. This method will be called for each request." (declare (ignore env)) (multiple-value-bind (res foundp) (dispatch (mapper this) (request-path-info *request*) :method (request-method *request*)) (if foundp res (not-found this))))
As you can see the
environment is being ignored here, and if you
look at the code of
DISPATCH function you will find the parts which
invoke our routes and pass down the parsed HTTP request params.
So, to fix this I’ve created a new sub-class of
lack:call method, but also dynamically binds the
environment, so that at least it is present when HTTP routes need
it. So, time to make things work. Fire up your REPL and load the
systems we will use.
(ql:quickload :ningle) (ql:quickload :lack) (ql:quickload :clack)
And now we will define our sub-class of
ningle:app and provide our
custom implementation of
lack:call. The important bits here are that
we are dynamically binding
(defparameter *request-env* nil "*REQUEST-ENV* will be dynamically bound to the environment context of HTTP requests") (defclass app (ningle:app) () (:documentation "Custom application based on NINGLE:APP")) (defmethod lack.component:call ((app app) env) ;; Dynamically bind *REQUEST-ENV* for each request, so that ningle ;; routes can access the environment. (let ((*request-env* env)) (call-next-method)))
Here’s an example middleware we are going to use, which pushes a message to the environment, which will later be used by our example route.
(defun my-middleware (app) "A custom middleware which wraps a NINGLE:APP and pushes additional metadata into the environment for HTTP routes. (lambda (env) (setf (getf env :my-middleware/message) "my middleware message") (funcall app env)))
Okay, that’s pretty much it. Time to create an app instance and test things out.
(defparameter *app* (make-instance 'app)) (setf (ningle:route *app* "/") (lambda (params) (declare (ignore params)) (getf *request-env* :my-middleware/message))) (defparameter *wrapped-app* (lack:builder #'my-middleware *app*)) (defparameter *server* (clack:clackup *wrapped-app*))
This should get your app started and listening on the default
port. Finally we can query our simple API and see if things work.
$ curl -X GET -vvv http://localhost:5000/ * Trying 127.0.0.1:5000... * Connected to localhost (127.0.0.1) port 5000 (#0) > GET / HTTP/1.1 > Host: localhost:5000 > User-Agent: curl/7.86.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Date: Thu, 01 Dec 2022 17:48:18 GMT < Server: Hunchentoot 1.3.0 < Transfer-Encoding: chunked < Content-Type: text/html; charset=utf-8 < * Connection #0 to host localhost left intact my middleware message
Great, things work as expected now and we are able to interact with
environment, so our middlewares can do their job. Ideally, this
should be fixed in
ningle itself, as it makes more sense to be that
way in my opinion.
You can find the full code here.
Next, I will be reviewing
snooze, before I make up my mind between
snooze. Till next time!