Notes on the Advanced Akka course

The Ad­vanced Akka course is provided by Typesafe and is aimed at teach­ing ad­vanced us­ages of Akka. The course cov­ers the ba­sics of Akka, Re­mot­ing, Clus­ter­ing, Routers, CRDTs, Cluster Shard­ing and Akka Per­sist­ance. The fol­low­ing post starts with a gen­eral in­tro­duc­tion to Akka and presents the takeaways from the course as we ex­per­i­enced them.

A general overview of Akka

#

The reader which is already fa­mil­iar with Akka can skip this sec­tion.

Ac­cord­ing to the Akka site this is Akka:

Akka is a toolkit and runtime for build­ing highly con­cur­rent, dis­trib­uted, and fault tol­er­ant event-driven ap­plic­a­tions on the JVM.

Akka achieves this by us­ing Act­ors.

Act­ors are very light­weight con­cur­rent en­tit­ies.

Each Actor has a cor­res­pond­ing mail­box stored sep­ar­ately from the Actor. The Act­ors to­gether with their mail­boxes reside in an ActorSystem. Ad­di­tion­ally, the ActorSystem con­tains the Dis­patcher which ex­ecutes the hand­ling of a mes­sage by an actor. Each Actor only handles a single mes­sage at a time.

In Akka everything is re­mote by design and philo­sophy. In prac­tice this means that each Actor is iden­ti­fied by its ActorRef. This is a ref­er­ence to the actor which provides Loc­a­tion Trans­par­ency.

Act­ors com­mu­nic­ate with each other by send­ing mes­sages to an an­other Actor through an ActorRef. This send­ing of the mes­sage takes vir­tu­ally no time.

In ad­di­tion to ActorRef there ex­ists also an ActorSelection which con­tains a path to one or more act­ors. Upon each send­ing of the mes­sage the path is tra­versed un­til the actor is found or when not. No mes­sage is send back when the actor is not found how­ever.

States: Star­ted - Stopped - Ter­min­ated If an actor enters the Stopped state it first stops its child act­ors be­fore en­ter­ing the Terminated state.

Best-practices

#

Im­port the context.dispatcher in­stead of the global Scala ExecutionContext. It is the ExecutionContext man­aged by Akka. Us­ing the global con­text causes the Act­ors to be run in the global Thread pool.

You should not use PoisonPill as it will be re­moved from fu­ture ver­sions of Akka since it is not spe­cific enough. Roll your own mes­sage to make sure the ap­pro­pri­ate ac­tions for grace­ful shut­down are done. Use context.stop to stop your actor.

Place your busi­ness lo­gic in a sep­ar­ate trait and mix it in to the actor. This in­creases test­abil­ity as you can eas­ily unit test the trait con­tain­ing the busi­ness lo­gic. Also, you should put the cre­ation of any child act­ors in­side a sep­ar­ate method so the cre­ation can be over­rid­den from tests.

Remoting

#

With the Re­mot­ing ex­ten­sion it is pos­sible to com­mu­nic­ate with other Actor Sys­tems. This com­mu­nic­a­tion is of­ten done through ActorSelections in­stead of ActorRef.

Re­mot­ing uses Java seri­al­isa­tion by de­fault which is slow and fra­gile in light of chan­ging defin­i­tions. It is pos­sible and re­com­men­ded to use an­other mech­an­ism such as Google Pro­to­buf.

Clustering

#

Akka has a simple per­spect­ive on cluster man­age­ment with re­gards to split-brain scen­arios. Nodes be­come dead when they are ob­served as dead and they can­not re­sur­rect. The only way a node can come up again is if it re­gisters it­self again.

When a net split hap­pens the other nodes are marked as un­reach­able. When us­ing a Singleton, this means that only the nodes that can reach the singleton will ac­cess it. The oth­ers will not de­cide on a new Singleton in or­der to pre­vent a split-brain scen­ario.

An­other meas­ure against split-brain is con­tact­ing the seed nodes in or­der. The first seed node is re­quired to be up.

The seed nodes are tried in or­der.

FSM

#

There is an lib­rary for writ­ing fi­nite state ma­chines called FSM. For lar­ger act­ors it can be use­ful to use the FSM. Oth­er­wise stick to pure become and unbecome.

FSM also has an in­ter­val timer for schedul­ing mes­sages. How­ever, the use of stay() re­sets the in­ter­val timer there­fore you could have is­sues with never ex­ecut­ing what is at the end of the timer.

Routers

#

There are two dif­fer­ent kinds of routers: Pools and Groups. Pools are in charge of their own chil­dren and they are cre­ated and killed by the pool. Groups are con­figured with an ActorSelection that defines the act­ors to which the group should sent its mes­sages. There are sev­eral im­ple­ment­a­tions: Con­sist­ent Hash, Ran­dom, Round Robin, BroadCast, Scat­ter - Gather First, and Smal­lest Mail­box. The names are self-explanatory.

Synchronisation of data with CRDTs

#

Syn­chron­ising data between mul­tiple nodes can be done by choos­ing your data­type so that If the timestamps and events are gen­er­ated in one place no du­plic­ate entries oc­cur. There­fore mer­ging a map from a dif­fer­ent node in your map is eas­ily done by copy­ing entries you don’t already have to your own data.

This can be im­ple­men­ted by let­ting each mem­ber node broad­cast which data-points they have. Each node can then de­tect which in­form­a­tion is lack­ing and re­quest the spe­cific data from the node that claimed to have the data. At some fu­ture point in time all nodes will be in sync. This is called even­tual con­sist­ency.

Singleton

#

If you have a singleton cluster man­ager proxy it only starts when the cluster is formed. A cluster is formed if a mem­ber con­nects. The proxy will then pass on the buf­fered mes­sages.

Cluster Sharding

#

Shard­ing is a way to split up a group of act­ors in a cluster. This can be use­ful if the group is too large to fit in the memory of a single ma­chine. The Cluster Shard­ing fea­ture takes care of the par­ti­tion­ing of the act­ors us­ing a hash you have to define with a func­tion shardResolver. The sharded act­ors can be mes­saged with an unique iden­ti­fier us­ing ClusterSharding(system).shardRegion("Counter") which prox­ies the mes­sage to the cor­rect actor. ClusterSharding.start is what the Man­ager is to Singletons.

It is re­com­men­ded to put the shard­ing func­tions into a singleton ob­ject for easy re-use of your shards, con­tain­ing the func­tions to start the shard­ing ex­ten­sion and proxy to the shard etc. It is also con­veni­ent to adds tell and initialise helper func­tions to re­spect­ively send a mes­sage and ini­tial­ise the actor by its unique id.

Akka Persistence

#

Akka per­sist­ence uses a Journal to store which mes­sages were pro­cessed. One of the sup­por­ted stor­age mech­an­isms is Cas­sandra. It is also pos­sible to use a file-based journal which, of course, is not re­com­men­ded.

In the cur­rent ver­sion of Akka there are two ap­proaches to per­sist­ence: com­mand sourcing and event sourcing. Simply but, in com­mand stor­ing each mes­sage is first per­sisted and then offered to the actor to do as it pleases whereas in event sourcing only the res­ults of ac­tions are per­sisted. The lat­ter is pre­ferred and will be the only re­main­ing method in fol­low­ing ver­sions.

Both meth­ods sup­port stor­ing a snap­shot of the cur­rent state and re­cov­er­ing from it.

Command Sourcing

#

The main prob­lem with com­mand sourcing lies in that all mes­sages are re­played. This in­cludes re­quests for in­form­a­tion from dead act­ors which wastes re­sources for noth­ing. Moreover, in case of er­rors, the last mes­sage that killed the actor is also re­played and prob­ably killing the actor again in the pro­ces.

Event Sourcing

#

With event sourcing one only stores state chan­ging events. Events are re­ceived by the receiveRecover method. Ex­ternal side-effects should be per­formed in the receive method. The code for the in­ternal side-effect of the event should be the same in both the receive and receiveRecover meth­ods. The actor or trait for this will be named PersistentActor.

Actor offloading

#

One can use Akka Per­sist­ence to “pause” long liv­ing act­ors, e.g. act­ors that have seen no activ­ity lately. This frees up memory. When the actor is needed again it can be safely re­stored from the per­sist­ence layer.

Tidbits

#

Akka 3 is to be re­leased “not su­per soon”. It will con­tain typed act­ors. The con­sequence of this is that the sender field will be re­moved from the actor. There­fore, for request-response, the ActorRef should be ad­ded to the re­quest it­self.

Concluding

#

The Ad­vanced Akka course gives a lot of in­sights and con­crete ex­amples of how to use the ad­vanced Akka fea­tures of clus­ter­ing, shard­ing and per­sist­ing data across mul­tiple nodes in or­der to cre­ate a sys­tem that really is highly avail­able, re­si­li­ent and scal­able. It also touches on the bleed­ing edge func­tion­al­it­ies, the ideas and con­cepts around it and what to ex­pect next in this grow­ing eco­sys­tem.