Notes on the Advanced Akka Course

| Comments

The Advanced Akka course is provided by Typesafe and is aimed at teaching advanced usages of Akka. The course covers the basics of Akka, Remoting, Clustering, Routers, CRDTs, Cluster Sharding and Akka Persistance. The following post starts with a general introduction to Akka and presents the takeaways from the course as we experienced them.

A general overview of Akka

The reader which is already familiar with Akka can skip this section.

According to the Akka site this is Akka:

Akka is a toolkit and runtime for building highly concurrent, distributed, and fault tolerant event-driven applications on the JVM.

Akka achieves this by using Actors.

Actors are very lightweight concurrent entities.

Each Actor has a corresponding mailbox stored separately from the Actor. The Actors together with their mailboxes reside in an ActorSystem. Additionally, the ActorSystem contains the Dispatcher which executes the handling of a message by an actor. Each Actor only handles a single message at a time.

In Akka everything is remote by design and philosophy. In practice this means that each Actor is identified by its ActorRef. This is a reference to the actor which provides Location Transparency.

Actors communicate with each other by sending messages to an another Actor through an ActorRef. This sending of the message takes virtually no time.

In addition to ActorRef there exists also an ActorSelection which contains a path to one or more actors. Upon each sending of the message the path is traversed until the actor is found or when not. No message is send back when the actor is not found however.

States: Started - Stopped - Terminated If an actor enters the Stopped state it first stops its child actors before entering the Terminated state.

Best-practices

Import the context.dispatcher instead of the global Scala ExecutionContext. It is the ExecutionContext managed by Akka. Using the global context causes the Actors to be run in the global Thread pool.

You should not use PoisonPill as it will be removed from future versions of Akka since it is not specific enough. Roll your own message to make sure the appropriate actions for graceful shutdown are done. Use context.stop to stop your actor.

Place your business logic in a separate trait and mix it in to the actor. This increases testability as you can easily unit test the trait containing the business logic. Also, you should put the creation of any child actors inside a separate method so the creation can be overridden from tests.

Remoting

With the Remoting extension it is possible to communicate with other Actor Systems. This communication is often done through ActorSelections instead of ActorRef.

Remoting uses Java serialisation by default which is slow and fragile in light of changing definitions. It is possible and recommended to use another mechanism such as Google Protobuf.

Clustering

Akka has a simple perspective on cluster management with regards to split-brain scenarios. Nodes become dead when they are observed as dead and they cannot resurrect. The only way a node can come up again is if it registers itself again.

When a net split happens the other nodes are marked as unreachable. When using a Singleton, this means that only the nodes that can reach the singleton will access it. The others will not decide on a new Singleton in order to prevent a split-brain scenario.

Another measure against split-brain is contacting the seed nodes in order. The first seed node is required to be up.

The seed nodes are tried in order.

FSM

There is an library for writing finite state machines called FSM. For larger actors it can be useful to use the FSM. Otherwise stick to pure become and unbecome.

FSM also has an interval timer for scheduling messages. However, the use of stay() resets the interval timer therefore you could have issues with never executing what is at the end of the timer.

Routers

There are two different kinds of routers: Pools and Groups. Pools are in charge of their own children and they are created and killed by the pool. Groups are configured with an ActorSelection that defines the actors to which the group should sent its messages. There are several implementations: Consistent Hash, Random, Round Robin, BroadCast, Scatter - Gather First, and Smallest Mailbox. The names are self-explanatory.

Synchronisation of data with CRDTs

Synchronising data between multiple nodes can be done by choosing your datatype so that If the timestamps and events are generated in one place no duplicate entries occur. Therefore merging a map from a different node in your map is easily done by copying entries you don’t already have to your own data.

This can be implemented by letting each member node broadcast which data-points they have. Each node can then detect which information is lacking and request the specific data from the node that claimed to have the data. At some future point in time all nodes will be in sync. This is called eventual consistency.

Singleton

If you have a singleton cluster manager proxy it only starts when the cluster is formed. A cluster is formed if a member connects. The proxy will then pass on the buffered messages.

Cluster Sharding

Sharding is a way to split up a group of actors in a cluster. This can be useful if the group is too large to fit in the memory of a single machine. The Cluster Sharding feature takes care of the partitioning of the actors using a hash you have to define with a function shardResolver. The sharded actors can be messaged with an unique identifier using ClusterSharding(system).shardRegion("Counter") which proxies the message to the correct actor. ClusterSharding.start is what the Manager is to Singletons.

It is recommended to put the sharding functions into a singleton object for easy re-use of your shards, containing the functions to start the sharding extension and proxy to the shard etc. It is also convenient to adds tell and initialise helper functions to respectively send a message and initialise the actor by its unique id.

Akka Persistence

Akka persistence uses a Journal to store which messages were processed. One of the supported storage mechanisms is Cassandra. It is also possible to use a file-based journal which, of course, is not recommended.

In the current version of Akka there are two approaches to persistence: command sourcing and event sourcing. Simply but, in command storing each message is first persisted and then offered to the actor to do as it pleases whereas in event sourcing only the results of actions are persisted. The latter is preferred and will be the only remaining method in following versions.

Both methods support storing a snapshot of the current state and recovering from it.

Command Sourcing

The main problem with command sourcing lies in that all messages are replayed. This includes requests for information from dead actors which wastes resources for nothing. Moreover, in case of errors, the last message that killed the actor is also replayed and probably killing the actor again in the proces.

Event Sourcing

With event sourcing one only stores state changing events. Events are received by the receiveRecover method. External side-effects should be performed in the receive method. The code for the internal side-effect of the event should be the same in both the receive and receiveRecover methods. The actor or trait for this will be named PersistentActor.

Actor offloading

One can use Akka Persistence to “pause” long living actors, e.g. actors that have seen no activity lately. This frees up memory. When the actor is needed again it can be safely restored from the persistence layer.

Tidbits

Akka 3 is to be released “not super soon”. It will contain typed actors. The consequence of this is that the sender field will be removed from the actor. Therefore, for request-response, the ActorRef should be added to the request itself.

Concluding

The Advanced Akka course gives a lot of insights and concrete examples of how to use the advanced Akka features of clustering, sharding and persisting data across multiple nodes in order to create a system that really is highly available, resilient and scalable. It also touches on the bleeding edge functionalities, the ideas and concepts around it and what to expect next in this growing ecosystem.

The Difference Between Shallow and Deep Embedding

| Comments

Deep and shallow embedding are terms associated with Domain Specific Languages (DSL). A DSL is a language geared toward a specific domain. The dot language is an example of such a DSL for describing Graphs. Conceptually, a shallow embedding captures the semantics of the data of the domain in a data type and provides a fixed interpretation of the data, whereas a deep embedding goes beyond this and captures the semantics of the operations on the domain enabling variable interpretations.

We will illustrate this difference by embedding a simple expression language with summation, multiplication and constants in Haskell. Haskell is especially well-suited for and often used as a host language for embedded DSLs.

We express our language with the following interface. A type synonym Exp for normal Ints and three separate functions representing summation, multiplication, and constants.

1
2
3
4
5
type Exp = Int

plus  :: Exp -> Exp -> Exp
times :: Exp -> Exp -> Exp
const :: Int        -> Exp

We embedded the data of the domain in Haskell and provided functions for construction of the model and we can easily represent the calculation of an expression as $4 + 6 * 8$ with the following lines of Haskell:

1
val = const 4 `plus` (const 6 `times` const 8)

The advantage of this embedding that calculating the value of our expression is very fast. Other than the value we cannot determine anything else regarding our expression. This becomes more problematic when we add variables to our language.

We change our type to contain binding information and add two functions to represent the assignment and usage of variables.

1
2
3
4
type Exp = ([String  Int], Int)

assign :: String -> Int -> Exp
var    :: String        -> Exp

And in our naivity we can write the expression $x + 6 * 8$ as follows:

1
val = var "x" `plus` (const 6 `times` const 8)

Obviously, evaluating this creates havoc! What is the value of x? We should, of course, have introduced it first:

1
val = let "x" 4 (var "x" `plus` (const 6 `times` const 8))

Now we have assigned a value to x and we can safely use it in our expression.

Had we used a deep embedding we could have prevented the cataclysmic error by first checking whether each variable is assigned before it is used. We create a deep embedding of our expression by using a Haskell data type.

1
2
3
4
5
6
data Exp where
  Plus   :: Exp -> Exp    -> Exp -- plus
  Times  :: Exp -> Exp    -> Exp -- times
  Const  :: Int           -> Exp -- const
  Assign :: String -> Int -> Exp -- assign
  Var    :: String        -> Exp -- var

Note that we do not specify how the bindings should be stored, only that such a thing exists. We now define a function that checks whether we use a variable before it is defined.1

1
2
3
4
5
6
7
8
9
useBeforeDefine :: Exp -> Bool
useBeforeDefine e = f []
  where
  f :: [String] -> Exp -> Bool
  f (Plus  l r) env      = useBeforeDefine l env || useBeforeDefine r env
  f (Times l r) env      = useBeforeDefine l env || useBeforeDefine r env
  f (Const _)   _        = False
  f (Assign var _ e) env = useBeforeDefine e (var : env)
  f (Var var)        env = not (var `elem` env)

With the function above we can check whether an expression is well-formed. With our deep embedding we can even define transformations of our expression; e.g. differentiate with respect to a variable.

1
2
3
4
5
6
7
diff :: Exp -> String -> Exp
diff (Plus  l r) dx      = diff l dx `Plus` diff r dx
diff (Times l r) dx      = (diff l dx `Times` r) `Plus` (l `Times` diff r dx)
diff (Const _)   _       = Const 0
diff (Assign var x e) dx = Assign var x (diff e dx)
diff (Var var)        dx | var == dx = Const 1
                         | otherwise = Const 0

Deep embedding allows us to utilize the semantics of our model by defining multiple interpretations of our DSL. The downside is that just calculating the value of our expression has become slower due to the added overhead of the constructors, whereas the shallow embedding can be evaluated by only using Ints.

In short:

  • Shallow embedding should be used when you only need a single interpretation or when you are in a hurry.
  • Deep embedding should be used in all other cases.

More reading material on this subject:

  1. Most often you should use folds (2) instead of this direct recursion.

Combining Graphviz (Dot) and TikZ With Dot2tex

| Comments

We all want to create good looking documents and good looking documents need good looking images. Because we want consistency and because we are lazy we want to do this as automatic as possible. That is why we use LaTeX, it creates beautifully typeset documents without much manual effort.

Similarly, we use graphviz to generate our graphs for us. It’s automatic layout is the best in the field and the (declarative) dot language is easy to understand and compact to write. We can either include the PDFs dot generated in our document by using \includegraphics or we could use the latex graphviz package, remember that we are lazy. We can easily get the image in our first example in our PDF.

Image with graphviz
Figure 1. Example of an image generated by Graphviz/dot

There is a shadow side to using Graphviz/dot as well. There are two problems. Firstly, the image just looks a bit out of place around the nicely smoothed text in a PDF. Secondly, we lack the ability to use TeX code in our graph. This means we are limited to the formatting by dot and the graphs could therefore appear out of style with other figures in our document.

No worries, with TikZ it is possible to create very fancy graphs and images in general but you have to all the positioning manually! Imagine inserting a node and having to reorder everything!

Enter dot2tex it brings all the love of graphviz/dot to TeX/TikZ. Using dot2tex has many advantages:

  1. Lets you write your graphs in familiar dot syntax;
  2. Let dot – or whichever layout engine you prefer – determine the placement of your nodes and arrows;
  3. Style your nodes however you want by using TikZ styles;
  4. Optionally, fine-tune the graph by adding extra tikz drawings.

Rather than manually calling dot2tex for every dot file you have please use the dot2texi package. This is the interface to dot2tex and when used as follows generates the image as displayed in Figure 2.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
  \begin{tikzpicture}[>=latex',scale=0.5]
    % set node style

    \begin{dot2tex}[dot,tikz,codeonly,styleonly,options=-s -tmath]
        digraph G  {
            node [style="n"];
            p [label="+"];
            t [texlbl="\LaTeX"];
            6
            8
            10-> p;
            6 -> t;
            8 -> t;
            t -> p;
            {rank=same; 10;6;8}
        }
    \end{dot2tex}.
    \begin{pgfonlayer}{background}
        \draw[rounded corners,fill=blue!20] (6.north west) -- (8.north east) -- (t.south east)--cycle;
    \end{pgfonlayer}
\end{tikzpicture}
Example of TeX typesetting + TikZ background
Figure 2. Example of TeX typesetting + TikZ background

For more TikZ goodness check out the example site.

Happy writing!

Why You Should Switch to Declarative Programming

| Comments

We are reaching limits of what is feasible with imperative languages and we should move to declarative languages.

When applications written in imperative languages grow, the code becomes convoluted. Why? Imperatively programmed applications contain statements such as if X do Y else do Z. As Y and Z contain invisible side-effects the correctness of the program relies on some implicit invariant. This invariant has to be maintained by the programmer or else the code will break. Thus each time a new feature is added to an application or a bug is fixed the code for the application gets more complex as keeping the invariant intact becomes harder. After a while the code becomes spaghetti-code and bugs are introduced as the programmer fails to maintain the invariant. This is going to happen despite the best intentions of the programmer to keep things clean. Why is this?

Software and Building Architecture Compared

| Comments

People tend to only understand what they can see. For most people it is difficult to grasp more abstract matters without somehow visualizing them. Software is an example of such an abstract matter. Let us visit the process of developing software through a comparison with developing a building.

JCU App Installation Script

| Comments

With this script it should be possible to install and build the JCU app in a local directory. It does not build the UHC for you. If that is wanted the option could be build in of course!

Getting Rid of Programming JavaScript With Haskell

| Comments

For my Experimentation Project at Utrecht University I ported the “JCU” application to Haskell. The JCU application is used to give Dutch High school students the opportunity to taste Prolog.

The project uses the Utrecht Haskell Compiler and its JavaScript backend. The UHC translates Haskell to Core and then translates this Core language to JavaScript. For more information on this see the blog of the creator of the UHC JavaScript backend.

Please read my report on this project. The project is hosted on GitHub in the following repositories:

Incomplete proof in the HS JCU App Complete proof in the HS JCU App

update 28-01-2012: The keyword jscript in the UHC has been changed to js in order to avoid association with Microsoft’s JScript. Also new Object syntax is now available in the foreign import directives.

1
2
foreign import js "new Object()"
  newObject :: IO (JSPtr a)

A Transition to Static Site Generation

| Comments

Today I’ve launched my new blog. It is based on Octopress and works by statically generating the pages and then syncing them with the server.

If you are for example on OS X Lion and installed XCode 4.2 and you run into weird errors like a missing gcc-4.2, and Homebrew throws errors like this:

Error: The linking step did not complete successfully The formula built, but is not symlinked into /usr/local

Please install the gcc package from this nice fellow: osx-gcc-installer

And if you are getting nagged by rb-fsevent. Change

Gemfile
1
  gem 'rb-fsevent'

to

Gemfile
1
  gem 'rb-fsevent', "0.9.0.pre4"

Update The comments have been exported with the Wordpress plugin to Disqus. I’m currently looking at how to highlight code within Disqus comments.

Caching Hackage

| Comments

On several occasions I noticed that when performing a cabal update that the index was being downloaded at the rate of plus min 300 KB/s. Finally I got around to do something about this. I’ve set up a caching server located in Utrecht, The Netherlands. It is a caching proxy for the hackage repository. If you want to use it, add the following to your ~/.cabal/config file. (Or equivalent on Windows.)

1
remote-repo: cache:http://spockz.nl:12080/packages/archive

Be sure to comment out the already existing remote-repo. Otherwise, cabal will download both indexes and merge them, and we don’t want this.

The funny bit

Apparently this only helps if your machine is fast enough to process the index (untarring and all extra administration cabal performs).

Plainly getting the file from the cache:

1
2
3
4
> wget http://spockz.nl:12080/packages/archive/00-index.tar.gz
Saving to: 00-index.tar.gz
100%[======================================>] 3.394.821   10,9M/s   in 0,3s
2011-11-13 00:12:35 (10,9 MB/s) - 00-index.tar.gz saved [3394821/3394821]

And running cabal update with my cache as source:

1
cabal update -v3  7,50s user 0,21s system 99% cpu 7,736 total

And then finally, with the original repository:

1
cabal update -v3  7,57s user 0,25s system 28% cpu 27,372 total

So here we see that the user time is roughly the same but you spent almost three times more seconds waiting for your coffee to get cold. Any further speed improvements for cabal update will probably require optimalisation of the code.

The caching server

I’m using Varnish to cache the request to hackage. And here is my config file. Please shoot if you see any improvements.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
# This is a basic VCL configuration file for varnish.  See the vcl(7)
# man page for details on VCL syntax and semantics.

backend backend_0 {
	.host = "hackage.haskell.org";
	.port = "80";
	.connect_timeout = 0.4s;
	.first_byte_timeout = 300s;
	.between_bytes_timeout = 60s;
}


acl purge {
  "localhost";
	"hackage.haskell.org";
}

sub vcl_recv {
    set req.grace = 120s;
    set req.backend = backend_0;

		set req.http.host = "hackage.haskell.org";

    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
            error 405 "Not allowed.";
        }
        return(lookup);
    }

    if (req.request != "GET" &&
        req.request != "HEAD" &&
        req.request != "PUT" &&
        req.request != "POST" &&
        req.request != "TRACE" &&
        req.request != "OPTIONS" &&
        req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return(pipe);
    }

    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return(pass);
    }

    if (req.http.If-None-Match) {
        return(pass);
    }

    if (req.url ~ "createObject") {
        return(pass);
    }

    remove req.http.Accept-Encoding;

    return(lookup);
}

sub vcl_pipe {
    # This is not necessary if you do not do any request rewriting.
    set req.http.connection = "close";

}

sub vcl_hit {

    if (req.request == "PURGE") {
        purge_url(req.url);
        error 200 "Purged";
    }

    if (!obj.cacheable) {
        return(pass);
    }
}

sub vcl_miss {

    if (req.request == "PURGE") {
        error 404 "Not in cache";
    }

}

sub vcl_fetch {
    set beresp.grace = 120s;

    if (!beresp.cacheable) {
        return(pass);
    }
    if (beresp.http.Set-Cookie) {
        return(pass);
    }
    if (beresp.http.Cache-Control ~ "(private|no-cache|no-store)") {
        return(pass);
    }
    if (beresp.http.Authorization && !beresp.http.Cache-Control ~ "public") {
        return(pass);
    }

}

sub vcl_deliver {

}