Tag Archives: fp

Variance expression in functional programming

This time we’re going to explore how functional programs express variance. We’ll consider the same case as we did for TDA – getting a stored value from a Map, and storing a new value in a Map.

Retrieval

lookup :: Ord k => k -> Map k a -> Maybe a

In Haskell, the variance is expressed in the return type.

data Maybe a = Nothing
             | Just a

We might express this as follows:

variance ⊂ return type

Storage

We want to put a new value into the map.

Once again, the entropy of this function is two, see the previous variance post for further discussion.

Here’s the comprehensive version:

insertLookupWithKey :: Ord k => (k -> a -> a -> a) -> k -> a -> Map k a -> (Maybe a, Map k a)

This function actually allows the caller to do more than our original requirement, so a little explanation of that type signature is perhaps necessary.

(k -> a -> a -> a)

The caller passes in this function, which takes the key, the new value and the old value and produces the value that will be stored (most implementations will simply return the new value, rather than performing a calculation, mind).

Data.Map will only call this function if a value is already present – this information is present in the last part of the type signature:

k -> a -> Map k a -> (Maybe a, Map k a)

This is the part we’re going to focus on. Here we pass the key, new value, and the map to operate on. Note that the return type contains both the possibility of an old value (the presence of which tells us our value transforming function was called) and the new map.

Diversion one: encapsulation in FP

It’s starting to look as though functional programs eschew encapsulation in favour of type signature totality (definitely a thing). This is not true.

  • What if Maybe didn’t export its two constructors?

I fear the answer for some may be “huh”? So, a brief explanation of how this might work.

Haskell makes it relatively simple to obscure your implementation details. At the moment, we know that:

Maybe a = Just a
        | Nothing

In fact though, we only get to know this because the original author allowed those constructors out to play. It would be just as valid to export two functions that did the same but obscure the constructors, like so:

module Maybe (Maybe,just,nothing) where

data Maybe a = Just a
             | Nothing

just :: a -> Maybe a
just = Just
nothing :: Maybe a
nothing = Nothing

instance Functor Maybe where
fmap f (Just a) = Just $ f a
fmap _ Nothing = Nothing

Now, at the point of invocation, we can’t pattern match any more. We have to use the fact that Maybe is an instance of Functor in order to do anything with the value inside; there’s no way at all for us to get it out.

Example: reversing the value, if it exists.

fmap reverse $ lookup "key" map

So, it looks like the core of the FP world; Functors and Monads are very interested in not letting us execute a function on a value directly; you pass the function you would like executed to the enclosing context, and it can choose (based on the value inside) how to execute it.

In particular in Haskell, syntactic sugar (do,<-, etc) is provided to make it appear to the eye that this is not the case. That’s only for humans, though; in reality the value never escapes.

Uh, so what now?

We used the term “return type” on the assumption that it is the last thing we see in the type signature of a function. This is either almost true or almost completely false.

Tony Morris tells us that all Haskell functions take one argument, and, once you’ve read that link, I’ll hope you’ll agree he is right.

This means that for a function which appears to take n arguments there are in fact n - 1 return types. Or, perhaps, they have one return type, and it’s everything we see after the first ->.

We’ll stick with:

variance ⊂ return type

Totalitarianism, dogma and consequences

That’s enough pretentious subheadings -Ed

Just as we distilled OOP into a very dogmatic TDA, we’ve taken a very dogmatic view of what FP is. Effects are delayed until the last possible moment, and mutable state is excised.

Oddly, unlike in the TDA case, our chosen implementation language, Haskell, is very keen on enforcing this dogma. Attempting to perform calculations or effects outside of a function’s type signature is* a compile error.

From here on out, we’ll assume dogmatic FP === FP and that non dogmatic usages of FP languages are charlatanic. This isn’t necessarily true, but it will save us a few words here and there.

(*almost always is. There is always Debug.trace)

Diversion 2: A mirror image

Cast your mind back to our purified TDA map insert. What would it look like if we translate it to Haskell?

put :: (Ord k) => Map k a -> k -> a -> (a -> IO()) -> IO() -> IO()

You can think of a -> IO() as being a bit like Consumer and IO() as being like Runnable. Haskell’s lazy predisposition allows us to pass around putStrLn "nothing here" as an expression without executing it.

Let’s rename that, and fiddle with argument order:

tda_insert :: (Ord k) => k -> a -> Map k a -> (a -> IO()) -> IO() -> IO()

Let’s also consider the core of the FP signature:

fp_insert :: (Ord k) => k -> a -> Map k a -> (Maybe a, Map k a)

Now, recall our stricter, non pattern matching Maybe implementation from earlier. This told us that Maybe is actually a context for executing functions that take an a. It also provides the following:

maybe :: b -> (a -> b) -> Maybe a -> b

If we just plug b = IO() into that, we get:

maybe :: IO() -> (a -> IO()) -> Maybe a -> IO()

and with some further shuffling, it becomes this:

maybe_ish :: Maybe a -> IO() -> (a -> IO()) -> IO()

Now, if we elide Map k a from fp_insert (thus making the state change implicit)

fp_insert_ish :: (Ord k) => k -> a -> Map k a -> Maybe a

…and now, we almost have

tda_insert = maybe_ish . fp_insert_ish

This is very interesting! It feels like we’ve come across a little piece of symmetry between the two approaches.

One way to think of this, perhaps, is that our TDA insert has inlined Maybe. It acts both as an associative container and as a context for function execution. The only difference (mind out though, it’s a big one) is that fp_insert is far stricter about what those functions are allowed to do.

We perhaps shouldn’t be surprised at this; what we’ve really found is an artifact of tight encapsulation, which we introduced by definition in both cases.

Why compose isn’t quite working

fp_insert_ish, as far as . is concerned, has type

fp_insert_ish :: (Ord k) => k -> (a -> Map k a -> Maybe a)

and maybe_ish is really wanting a Maybe a to start with. One can bind enough of fp_insert_ish‘s parameters to get this to work; we’ll not spend the time doing that here.

Having found some symmetry (admittedly with a little bit of hand waving), let’s consider a less friendly asymmetry.

Interactions between FP and OOP

A lot of hot air has been generated about the exciting possibility of combining the powers of OOP and FP approaches, as if they are some sort of transformer robots that get a attack bonus, amazing music and lens flares whenever they touch.

It’s not particularly clear whether this is true. We won’t try to answer this question here at all. All we can do here is look at TDA‘s interaction with FP.

  • Can FP code call TDA code?

No

The TDA approach deliberately obscures some variance in its type signatures. You’d have to do some serious gymnastics to express the same thing in Haskell, and, once the core is infected with mutability, the IO() virus spreads inexorably outwards.

  • Can TDA code call FP code?

Just about.

One can start to preach the church of return type in the core of an application and gently push it outwards (TDA style callers can update a mutable reference to the newly returned FP core, then send out the (also returned) events).

This will cause some dissonance, but, and this is the very important point, only at changeover points. Purity, unlike mutability, isn’t upwardly mobile.

Diversion 3: Not today…

Whether this has any bearing on OOP / FP interweaving is left temporarily as an exercise to the reader.

Serendipitously, Eric Meijer just wrote something excellent on the wider question of hybrid OOP/FP approaches.

Conclusion

We found a tiny Maybe hidden in our TDA code; a symmetry due to the encapsulation shared between our two approaches.

One wonders if other exercises TDAing the living daylights out of other data structures would uncover any other fundamental types.

We found also one asymmetry between TDA and FP; implicit mutation simply isn’t permitted in FP, so we can nest the approaches in one direction only.

Extremely satisfying summary

Once we start to mutate, anyone above must mutate with us.

So: Pick the point of no return very carefully.

Next time

A selection from:

  • Do our truths about TDA tell us anything about OOP?
  • Does our TDA world bring us any new problems?
  • Does our TDA world bring us any benefits?
  • Will we ever get to talking about the Actor model?

Post script

There may be a small hiatus over the next couple of months while I develop related material into a workshop for 2014’s SPA Conference ( This session is mine ).

A laziness trick to relax your code

So, this week on Distilled Derivatives, some more laziness. And also, exceptions. Everyone loves them, really useful things for letting you move error handling into a single location that they are. Now and again though, their misuse can really get in the way of getting something done.

Retrieve the content of the first functioning url

Given n urls in some ordered collection, get the content of the first “working” url, where working is defined to mean “has http code 200 and non empty response content”. Once you’ve found it, put the response body of the url into some local file.

This should be easy, thought I. How wrong I was.

First problem

All the java http libraries throw IOException. Ok, that’s not too bad, we can get past that with only a little evil. Here’s our first go.

  def withUrlConnection() : String = {
    for (url <- urls) {
      try
      {
        val actualUrl = new URL(url)
        val connection = actualUrl.openConnection().asInstanceOf[HttpURLConnection]
        return stringFromHttpUrlConnection(connection)
      }
      catch {
        case e: IOException => {}
      }
    }
    throw new RuntimeException("all failed")
  }

This isn’t too bad – we’re only teetering on the edge of breaking the “don’t use exception for control flow” rules, rather than falling into the abyss. We can simply take the resulting String and write it to a file (implementation elided). Oh, and in case you were curious, here’s the implementation of stringFromHttpURLConnection (we’ll need it again later).

N.B I’ve completely elided any closing of any connection everywhere in this article, because like the technique I’m trying to describe, I am lazy.

  val stringFromHttpUrlConnection: (HttpURLConnection) => String = { connection : HttpURLConnection =>
    Source.fromInputStream(connection.getInputStream).getLines().mkString("\n")
  }

The problem worsened, however, when some kind soul tried to help out by creating a function like the following:

Problem 2 – an unhelpful helper

  def withUnhelpfulFunction(path: String) {
    var done = false
    for (url <- urls) {
      try {
        if (!done) {
          downloadFile(url, path)
          done = true
        }
      } catch {
        case e: IOException => {}
      }
    }
  }

  def downloadFile(url: String, path: String) {
    // lobs url response body into path on success, IOException if not. Implementation too boring to include.
  }

At first glance this function looks like it might save us some code, but once you start talking about failure or multiple urls, we’re screwed. It’d be perfect if we wanted a one shot download that noisily failed, and that’s probably the use case the original author had in mind. In general the approach the function takes is a bit evil – it’s performing IO and letting us know we succeeded by not throwing an exception. That’s not the world’s greatest signal.

So, two quite ugly solutions with exceptions. Is there another way?

My hobby: getting lazy evaluation to write my program for me

Functional programming has given us some excellent tools to help decompose this problem, and there’s a neat lazy evaluation trick that can get us out of jail here. What do we need? A function that will:

  • synchronously resolve a url to the response body
  • include the possibility of failure in its return type

Roughly, this is equivalent to URL -> Maybe String (or Option[String], in Scala land). Try as I might, I can not find a scala or java http client library that follows this approach. Dispatch looks terrific but is also asynchronous, and I don’t really want to talk about futures in this post. Handily, numerous other sandbox projects I’ve tinkered with have already forced me into writing my own, so I have a handy function lying around.

  def withHttpLib() : String = {
    val requestor: HttpRequestor = new InsecureHttpRequestor()
    val results = urls.map(url => requestor.performGet(url, stringFromHttpUrlConnection))
    results.find(_.wasSuccessful()).getOrElse(throw new RuntimeException("all failed")).getResult
  }

Look ma, no for or if! Even better, this approach appears much shorter, however, this is mostly because much of the responsibility has moved into the little http library. Let’s walk through the execution of this program.

A little secret

  val urls = Stream("www.digihippo.net/home.html",
                    "www.digihippo.net/index.php",
                    "www.digihippo.net/foo.bar")

The structure the urls are sitting in is actually Scala’s excellent Stream abstraction. By default, operations on Streams are applied lazily. For example, if we invoked map on a particular instance of Stream, execution of the function we pass in is deferred until someone tries to extract an element from the Stream. In our earlier examples, we use Scala’s for comprehension to iterate across urls. In our favourite implementation, you can see that we’ve started applying map to it instead.

N.B These won’t actually work if you do new URL(url) on them. They’re in the current format because my library doesn’t yet work out whether to connect securely or not from the passed in url String.

What’s this http library doing then?

    val requestor: HttpRequestor = new InsecureHttpRequestor()

The requestor here is an object from my http library. It’s insecure because it doesn’t deal with https (it doesn’t lack confidence), just so you know. To execute a given request, one provides a url, and a response decoder – the requestor returns you a type parametrized Http object, which may contain the thing you wanted to decode, or an error. What’s happening underneath? Well, something like this:

  • Attempt to get url
  • On failure, return Http.failure() with some helpful details
  • On success, attempt to parse using provided response decoder
  • If response decoder succeeds, return whatever it decoded
  • If it failed, return Http.failure() with some helpful details

Hold on though – why does our decoder get the opportunity to return any sort of Http? Why doesn’t it just attempt a parse and give up? Well, we have to really think about what makes any given http request a failure. The definition in this post is not necessary universal – sometimes we expect a 404, or a 500, and still wish to parse something from the resulting connection.

As such, at the moment, the decoder can examine any aspect of the created HttpURLConnection and decide for itself, if it wants to, that this request was a failure. Our decoder does no such thing though – it just attempts to grab the response body and whack it into a string. The library will helpfully catch any exception resulting from playing with HttpURLConnection and return you an unsuccessful Http instance with the error information stashed inside.

What is this line really doing?

    val results = urls.map(url => requestor.performGet(url, stringFromHttpUrlConnection))

So, hands up – who thinks we’re firing off http requests as we invoke map here? No-one? Good. Glad to see people are keeping up. What we’re really doing here is creating a new Stream, which at the moment is actually a collection of functions that, when called, will return an Http[String]. So this line actually has no side effects, we’re really just defining how to compute the next elements of results, rather than computing them.

This is where the magic happens

    results.find(_.wasSuccessful()).getOrElse(throw new RuntimeException("all failed")).getResult

Now, the density of this line is considerably higher than the others. If that’s a problem for you, you might be a n00b. So, what the hell is happening?

Well, that find invocation would be better named findFirst. It traverses results, examining each element to see if it was a success. That means it needs to compute the element! So, find actually does this:

  • Execute function to obtain element. N.B this finally executes our map function
  • Examine result to see if it was successful
  • If not, repeat with next element

So, in fact, just by defining results in the way that we have means that find expresses the control flow of our application for us. One could legitimately criticize the first two solutions for repeating this standard library code, and, in fact, functional languages go out of their way to make extracting this sort of legwork easier. If you’re wondering why this is important, measure how many times you write try/catch/finally in java (or similar) next week.

The remainder of the line does the necessarily evil to fulfil something like the original contract, where we wanted a String that we could later write out to a file. We need to extract the actual value from the find result (find actually gives us back Option[Http[String]], because there’s no guarantee any of our results matched), throwing if none were successful, and then we need to extract the String out of the found successful request.

An apology

Purist functional programmers will likely be groaning at this point. The side effects of our little getter don’t happen at the top of our program, necessarily, and the usage of Option (and the Http type, which is pretty similar to Option) is not particularly idiomatic (hitting get like functions on monadic types is frowned upon).

Some side notes

Interestingly this very same function was implemented as part of a bash build script elsewhere, and it embodies a lot of the lessons that we’ve seen in the code here. Here’s a cut down equivalent.

#!/bin/bash

function noisy_curl() {
    echo "issuing request to get $1"
    return `curl -s -f $1`
}

noisy_curl http://www.digihippo.net/main.html ||
noisy_curl http://www.digihippo.net/index.php ||
noisy_curl http://www.digihippo.net/site.html

This is, given how much effort we’ve gone to set up the same effect in Scala, embarrassingly trivial. One can start to see the elegance of the unix way of doing one simple task, and writing the result to stdout, while also indicating success with a return code. Here we use the -f flag to tell curl to ‘fail’ if we get a suitably erroneous looking response code, and then just set up a series of curl invocations such that the first success short circuits the others. One starts to agree with all those people on the internet who think UNIX pipelining looks a helluva lot like composition as seen in functional programs.

We’ve come a long way baby.

So let’s stop (for now). We could certainly expand on our example by doing something more interesting that putting that result String into a file. We could consider approaches where we never parse the whole String but instead hold on to the InputStream instead. These deserve to be posts in their own right though, so we will not discuss them further here.

What did we learn?

  • Lazy evaluation allows you to elide boring control flow from your code. More generally, a functional approach can help you separate the valuable parts of your code from arduous syntactic toil
  • Wrapping exceptions into return types at the point where they are thrown makes life so much easier
  • Communicating errors in exceptions is really only good for one shot calling patterns

…and, again, more generally, exceptions are vastly overused. How often per day do you get a broken web page when browsing? Is it really exceptional? Are browsers dealing with 404 or 500 via an exception path? I seriously hope not. Exception abuse seems particularly prevalent in java, perhaps because so many libraries use IOException as their IO alarm, and possibly also because the language makes it so difficult to work with something like the Http type. Once again, I find myself thinking that Java 8 really cannot arrive soon enough.

Moral of the story

Stop throwing all that stuff around and relax, dude.