I was honoured enough to do another lightning talk at SkillsMatter.  I’m afraid I had easily the least interesting talk of the evening, as you can see from the hastily prepared slides on Google Docs.

Neale Swinnerton gave a great talk on Paredit.  I’m still trying to nail paredit, and this was a great help.  The slides are a thing of beauty, watch them in full screen mode. 

Nick Rothwell gave a lightning talk that I wish had been longer, on teaching Clojure to artists.  He’s done some amazing and thankless work on embedding Clojure in MaxMSP.  As always, he gave a great demo.

Malcolm Sparks has already written up the main talk, which was excellent.  I’m not convinced I agree with him about excluding executable code from configuration (I have form on this).  Ultimately, I believe that “configuration” is just a way we describe code properties that different between environments.  There’s no need for it to be XML, or an INI file, or RDF.  The only requirement really is for it to be findable and editable at short notice.

Neale is doing another talk on 6th March on Clojurescript One.

Anyone learning Clojure like I am is recommended to read the source code to Jay Field’s Expectations library.  I was particularly interested to read his article on scenarios.  There’s some very interesting ideas there, but at the moment I wanted something a bit more basic.  “with-redefs” already provides the ability to replace one function with another, specifically for mocking purposes.  So really all I wanted was the ability to capture function invocations.

So, here’s a working mechanism.  You use with-captures exactly the way you would use with-redefs, only you can call the “captures” function on the redefined function and it will return the list of previous function invocations.

The most interesting aspect of the code is the use of a record implementing IFn.  Implementing records well remains a bit verbose (well, by Clojure standards).  I’d highly recommend checking out defrecord2 by David McNeil, which is looking to simplify working with records.

(ns hendrix.test.capture)

(defn ignore
  "Ignores the inputs and returns the outputs.  Useful as mock target."
  [& args] nil)

; Capture

(defn capture-invoke [{:keys [f captures]} args]
  (let [r (apply f args)]
    (swap! captures conj args)))

(defrecord capture-t [f captures]
  clojure.lang.IFn
  (invoke [this] (capture-invoke this []))
  (invoke [this a] (capture-invoke this [a]))
  (invoke [this a b] (capture-invoke this [a b]))
  (invoke [this a b c] (capture-invoke this [a b c]))
  (invoke [this a b c d] (capture-invoke this [a b c d]))
  (applyTo [this args]
    (clojure.lang.AFn/applyToHelper this args)))

(defn new-capture [f]
  (new capture-t f (atom [])))

(defn to-capture [[v f]]
  (new-capture (if (= f :v) (var-get v) f)))

(defn to-capture-map [h]
  (zipmap (keys h) (->> h (map to-capture))))

(defn captures [c]
  (-> c :captures deref))

(defn with-captures-fn [bindings action]
  "Like with-redefs-fn, only you can call 'captures' on the redefined functions."
  (-> bindings
      to-capture-map
      (with-redefs-fn action)))

; Code ripped off from with-redefs
(defmacro with-captures
  "Like with-redefs, only you can call 'captures' on the redefined functions."
  [bindings & body]
  `(with-captures-fn ~(zipmap (map #(list `var %) (take-nth 2 bindings))
                              (take-nth 2 (next bindings)))
     (fn [] ~@body)))

(defn add-two [x] (+ x 2))

(defn example []
  (with-captures [identity :v
                  add-two ignore]
    (identity 3)
    (identity 6)
    (add-two 7)
    {:ignore (captures add-two)
     :passthrough (captures identity)}))

I was interested to read Paul Ingles article on uSwitch’s handling of SSL.  Particularly interested because I remember writing the code he is replacing (It was about seven years ago, I believe...).  I thought makes a good case study in how your language choice affects your code.  To recap his article, uSwitch’s architecture offloaded SSL to hardware.  This meant that detecting whether you were running under SSL was impossible without a custom header.  Furthermore, the SSL/Non-SSL functionality was a bit more complex than a simple “redirect insecure pages”.

However, after you’ve added the custom header to your load balances from a programming perspective your problems are only just starting.  The uSwitch code base at the time was ASP.NET 2.0.  To access the request, you had to use the Request object, which was sealed.  So you didn’t really have any other option than to define a method that tested for SSL and shout at anyone who forgot to use it.  uSwitch had a large number of developers and high turnover, which made things like this an on-going issue.

Solving in Ruby

Paul’s solution to this problem is pretty much the right thing to do in Ruby.  He modifies the behaviour of Rack in order to support his use case.  No-one else working on an unrelated part of the code base ever needs to know about this subtlety of the SSL implementation.  Thus, a dynamic typed language with mutable classes demonstrates massively better separation than the statically-typed C#.

So Ruby has taken the pain away in a page of code.  However, can we improve on this if we move to a dynamically typed language with immutable state?  Like my favourite language, Clojure.  Well, are there any problems left?  Yes, although admittedly they’re not huge:

• We’ve moved the problem of “make people call this custom function” to “make people call this standard function”.  If they’re used to old versions of the rack and examine env directly, they’ll circumvent it.  (Admittedly, it’s my belief that Forward’s developers are typically of higher quality than old-school uSwitch’s.)
• We’ve modified Rack to get work done, which is fine as long as no-one else does the same thing.
• We’ve actually had to duplicate some of the functionality from Rack to make sure it behaves the same way.

Solving in Clojure

Now, the only way to remove the first problem is to actually modify the request.  In languages with mutable state this is usually regarded as a bad thing, because it’s very hard to track exactly what could be affected by your change.  Immutability solves that.  Modifying the request also prevents us from having to modify the underlying architecture, so any solution we build will be fully composable with anything else we choose to use.

(defn non-standard-https [handler request]  
    (handler (if (get-in request :headers "HTTP_USWITCH_HTTPS")  
                 (assoc request :scheme :https)  
                 request)))  
                 
(def handled-routes (partial non-standard-https routes))

What has the Clojure version given us?  Because the request is immutable, only the routes we choose to affect take the functionality.  (Not really an issue in this particular case, but still a nice property to have.)  We've not had to modify Ring: its middleware is fully composable and because of this property, we haven't had to duplicate Ring functionality in our own code.  It’s not configurable, but at four lines, it hardly needs to be.

To recap:

• Handling this simple piece of functionality at uSwitch in C# was a constant code-quality issue.
• Handling it in Ruby pretty much fixes the problem.
• Handling it in Clojure makes it look like a non-problem.

I’m genuinely of the belief that the only thing Clojure’s web solution lacks is maturity.

TL;DR For all that people think of Clojure as a “hard” “propeller-head” language, it’s actually designed right from the start not for intellectual purity, but developer productivity.

I thought I’d share my recent experiences of developing a commercial Clojure application.  It’s currently racking in about 800 lines of Clojure and 2000 lines of CoffeeScript.  I avoided ClojureScript for the simple reason that I had no experience in it, so I’m afraid I have no great insights there.  I guess it’s worth pointing out that although this is the first time I’ve really used Clojure in anger, I’ve been studying it for two years and have spent weeks of time practicing.  I can recommend pretty much all of the main teaching tools, labrepl, Clojure koans and 4clojure.

So, what does the application do?  It’s a web tool for calculating and analysing various costs my bank incurs.  The calculation itself and the UI is pretty much entirely in CoffeeScript using Backbone and jQuery.  The 800 lines of Clojure perform the following functions:

• A REST back end for the data.
• Windows authentication
• Serving the front page, which is the entire application.
• User permissioning
• A quick asset pipeline that compiles the CoffeeScript on demand.
• Charting

Adding Excel export shouldn’t make the code base much larger, maybe another 100 lines.  I should add that most of those 800 lines aren’t long and a fair number of them are blank.  To give you an idea of just how expressive this is, I had an earlier back end written in C#.  800 lines wouldn’t get you to the end of the entity definitions and NHibernate mappings.

More Than Just Concise

I suspect even Clojure’s greatest detractors would agree that it’s terse.  However, the real win is just quite how productive it is.  The more I’ve worked with it, the more surprised I’ve been and just how effective it can be.  Whilst certain languages on the functional side such as OCaml, Racket and Haskell have communities that have many competing priorities, of which developer productivity is only one, Clojure has a laser focus on beating the averages.  The first and most significant of the productivity features is its status as a JVM language.  This gives you access to a huge number of libraries straight out of the box:

• Waffle
• POI
• JTDS (which I used for SQL Server and Sybase connectivity)
• JFreeChart

Without POI, Excel export would be just a CSV file.  Without Waffle, Windows Authentication would be impossible: that is to say, there’s no way I could justify the expense of getting it to work and I’d have been left with implementing my own authentication system.  JFreeChart wasn’t absolutely essential: I could probably have got away with drawing the images myself, but still my productivity was raised by having it there.  Finally, JTDS is rock solid and fast.  The database connectivity story of Java is light years ahead of anything you’ll see in Node,* never mind Haskell.

There is a catch to all of this: you have to read and work with Java.  This isn’t particularly hard, but it means you become very familiar with the wall of inanity.  (I keep thinking “seriously, how many layers of abstraction do you need?”.)  However, there are thoughtful features that enhance the Java experience.  .. gives you the ability to chain method calls with less ceremony than Java.  “doto” gives a the advantages of jQuery style chain syntax without any need to change the underlying methods.  Finally, the REPL is a real help.  Getting the exactly correct chart settings is much easier when you can adjust it on the fly.  (You’ll note that this is a scenario that TDD doesn’t help you with at all.)

Better Than A Better Java

After the ability to just use Java libraries, there comes the excellent Clojure libraries.  Leiningen is an excellent build tool.  clj-webdriver takes the pain out of Selenium programming.  Andrew Brehaut has already written a pretty definitive guide to the Clojure web stack.  I’ve had the ability to edit any file in my server and have the changes reflected immediately (providing the file compiled).**  It’s amazing how few web frameworks can actually do this.  For instance, ASP.NET can do it for views, but not models or controllers.

More than just the fact that they’re good is the sheer pleasure of interoperability.  A lot of this comes from the basic philosophy of composability.  When everything’s a function, sequence or a hashmap, connecting one module to another is pretty easy with very few dependencies.  It’s also eminently testable.  However, Clojure goes deeper than that with extremely thoughtful features.  For instance, I’m using incanter to generate charts.  They return JFreeChart objects.  Compojure defines a protocol called Renderable that allows you to specify how return types get converted into Ring responses.  Now, you could apply an adapter pattern in most languages to link these two up.  In Clojure, you can declare that a type implements a protocol and it just works.  You don’t need to extend the type; you don’t need to monkey patch it; you don’t need to introduce a proxy object under your control.

There are other features that make developing easier as well, such as macros, dynamic binding, software transaction memory and multimethods, that I haven’t even needed to use.  I’m left with a very readable code base that is sensibly structured and easily expandable.  Surprising as it may seem, I’ve actually found it even faster to use than node.js.

*Although Brian Carlson’s node-postgres is quite superb.

**Some of this I wrote myself, but frankly, it only took few hours, and that included figuring out how to integrate with Leiningen.

I did a lightning talk for the LCUG earlier this month.  The idea was to give an idea of the things you can do with closures and was principally aimed at Java programmers who are interested in Clojure.  These are the slides, these are the speaker’s notes and here’s the video taken by skillsmatter.  It come in at under six minutes.  I was the first “support act” for Sam Aaron, whose talk isn’t up since it was a dry run for his Clojure Conj talk.  However, you can also see Philip Potter talking about labrepl and Nick Rothwell talking about DJing with Clojure.

David Havelin was the guy who broke this news: you can now use CoffeeScript directly on windows.  Here’s what you need to do:

• git clone git://github.com/jashkenas/coffee-script.git
• cd coffee-script/bin
• Download the latest node.exe (http://nodejs.org/dist/v0.5.7/node.exe) into the same directory.  Note the is an “unstable” version, but you can’t get away with 0.4.x
• ./node coffee --compile c:\myfile.coffee

Finally, throw away all of your over-complex coffeescript on windows solutions.  Hopefully soon there will be a batch file.

It’s fairly well known that you can override Backbone.sync in order to provide your own backing store.*  However, what isn’t documented but is clearly visible in the source, is that this.sync not only exists but takes priority.  This is pretty vital if you ever want a presentation-model style collection, where adding and removing causes changes to other models in your application.  I’ll warn you that you need to step through the framework code fairly carefully to get this to work, but it’s really useful.

*Another useful tip: if you’re using ASP.NET MVC, add a route like this

routes.MapRoute("Delete", "{controller}/{id}", new { action : "DELETE" }, 
    new { httpMethod = new HttpMethodConstraint("DELETE"))

Because ASP.NET MVC doesn’t seem to be able to cope with just slapping an [HttpDelete] attribute on an Index method.

So, after my first article I decided to ask Mike Rettig what I’d missed.  I have to admit to still not having had time to play with it myself,* however his reply was sufficiently interesting I asked to be allowed to share it.  He was kind enough to permit this.  There’s no explicit quoting here, but pretty much the credit for the pain points section is owing to Mike.  Any errors are my own.

Publish/Subcribe Pain Points

In the original article I stated that client and server were identical.  This isn’t really true.  In particular, a server is an acceptor and a client is an initiator, in standard TCP style.  Although both can publish and subscribe, the presence of topics in the protocol allows for server side filtering.  To make all of this easier to understand, imagine you’re writing a simple application that prints up the last trade in a particular list of stocks.  In classic pub/sub as ZeroMQ implements, you’d connect up to a stream that published every trade as it went down.  This would mean that you were receiving many updates in which your client had no interest.  You’re flooding the network with stuff that doesn’t matter, and usually your network is your most limited resource (certainly, very few have to worry about processing power and storage anymore…).  Using the topic, the server can be made aware of the necessary filtering that needs to occur.

This brings us to the next strength of Jetlang Remoting that I missed, probably because I only read the protocol and not the code.  When you subscribe, the server receives an event telling you this has occurred.  This is actually really significant.  Let’s return to our example application.  Let’s say we’re interested in the trades of some obscure stock, call it Recondite Inc.  Now, Recondite is an extremely illiquid and only trades once an hour.  In the classic model, you will subscribe and only get told about the stock when the next trade occurs.  Pretty much everyone regards this as part and parcel of the way pub/sub works and the way around it is fairly well understood: you just get the current state via an RPC first.  However, with the Jetlang model, the server is not only aware of what interests you, the server is aware of when you began.  Therefore it can publish the initial state through the topic.

These aren’t obscure points.  Most systems I’ve worked with that involved any form a real-time data distribution encountered both of those pain points.

The last point that Mike made concerns versioning.  The topic nature of subscriptions makes it relatively easy to identify the version of the client.  In turn this enables the server to filter messages that it knows the client can’t handle.  Achieving this without server-side filtering is possible, but only by throwing ports at the problem.

Complex Subscriptions

Lastly, I made a point last time that topics could have been made byte arrays, for complex subscription, but in practice the fact that every reply contains the topic suggests you’ll want to keep the topic small.  An alternative, but more complex arrangement, would be for subscriptions to return a subscription ID first and then send data messages with the subscription ID.  Once you recognize that model, you can see that you could set up such topics using an RPC call first.  Jetlang therefore provides the flexibility to do complex subscriptions without requiring every program to implement code for it.

*I’m spending most of my time with client side UI and backbone.js at the moment, of which more another time.

Any man with a wife or girlfriend knows a fundamental fact: women can multi-task and men can’t.  V8 in this respect is exactly like a man: it can only do one thing at once.  Node.js uses V8 and makes a virtue of it.  Only doing one thing at once means there’s no race conditions in your memory model.  None.  I’ve lost count of the number of times I’ve been writing node.js code and worrying about “what happens if the array length changes while I’m reading” before realizing it 100% can’t happen.

One thing that men know is that you don’t actually need to do two things at once.  Consider how you make a cup of tea

kettle.boil()
cup.fill()
cup.waitUntilBrewed()
cup.serve()

Well, actually, if you want to do the dishes at the same time, this isn’t how you do it at all.  What you do is switch the kettle on, then you go and start washing.  When the kettle clicks, you then stop what you were doing and fill the cup.

kettle.switchOn()
kettle.on ‘boiled’, –> cup.fill()
cup.on ‘brewed’. –> cup.server()

Notice that we’ve lost the names “boil” and “waitUntilBrewed”.  These are words that denote processes.  Everything in node.js is asynchronous, and process verbs imply a sychronous approach.  The words we are left with indicate actions and events.  It’s also worth noticing that with a multi-threaded environment, you’d be very careful to ensure that event bindings were set up before you kicked off the process that could fire the event.  With node.js, you’re guaranteed that your current code will finish before anything else happens.  While you’re talking, you have node’s undivided attention until you stop talking.

Doing The Dishes

So, how do you do the dishes?  Well, the simplest way to handle this would be

_.forEach dishes, (d) –> d.wash()

If you do this, you’ll never hear the kettle boil.  Remember, a man can only do one thing at once, even pay attention.  So, you need a way to specify that you want to pay attention

interruptibleForEach = (list, f) –>
    index = 0
    action = –>
        return if index >= list.length
        f list[index]
        index++
        process.nextTick action
    action()

You might want to compare this code to the Caliburn Micro Co-Routine Trick.  It’s fundamentally the same code.  There’s a catch, though.  You won’t stop washing dishes for very long.  In fact, you’re guaranteed to wash a dish between filling the cup and it being brewed to your satisfaction.  Now, you could generalize all of this to interruptible processes, but you’d be missing the point: doing a synchronous process like washing dishes is a fundamentally bad idea in node.js.  If you can’t figure out a decent way to express it as a series of events, you need to hive it off to a sub-process.  You then communicate with that through asynchronous RPC or pub/sub.  I’m going to avoid any obvious gender stereotype humour here.  If you want that sort of stuff, you can read the ZeroMQ Guide.

Personal Note: I was hit by Hurricane Penelope six weeks ago.  She is insanely gorgeous and an awful lot of work.  As a consequence, it’s quite hard to get out blog articles, so please forgive me any stupid errors and formatting problems.  Next time, I hope to talk about Jetlang Remoting again, but it might take a while...

Creating long chains of transformation using the thrush operator looks attractive, but what if you need to see what it's doing?

Consider the following code:

(-> 1  
  inc  
  inc)

Now it's obvious that this returns 3, and that the intermediate value was 2, but the following code proves it:

(defn thread-println [v prefix]  
  (doto v (->> (str prefix) println)))

(-> 1  
  inc  
  (thread-println "Intermediate value was")  
  inc)

You'd need to write a second function for ->>, sadly.