FinagleCon

FinagleCon was held at TwitterHQ in San Fran­cisco. It is re­fresh­ing to see a nice work­ing at­mo­sphere with free food and drinks. Now for the con­tents.

Twitter’s RPC frame­work, Fin­agle, has been in pro­duc­tion since Au­gust 2010 and has over 140 con­trib­ut­ors. In ad­di­tion to Twit­ter, it has been ad­op­ted by many large com­pan­ies such as SoundCloud. Ini­tially writ­ten in Java with FP con­structs (mon­ads, maps, etc.) all over, it was soon after re­writ­ten in Scala.

Fin­agle is based on three core con­cepts: Sim­pli­city, Com­pos­ab­il­ity, and Sep­ar­a­tion of Con­cerns. These con­cepts are shown through three prim­it­ive build­ing blocks: Future, Service, and Filter.

• Futures provide an easy in­ter­face to cre­ate asyn­chron­ous com­pu­ta­tion and to model se­quen­tial or asyn­chron­ous data-flows.
• Services are func­tions that re­turn fu­tures, used to ab­stract away, pos­sibly re­mote, ser­vice calls.
• Filters are es­sen­tially dec­or­at­ors and are meant to con­tain mod­u­lar blocks of re-usable, non-business lo­gic. Ex­ample us­ages are LoggingFilter and RetryingFilter.

The use of Fu­tures makes it easy to test asyn­chron­ous com­pu­ta­tions. Ser­vices and fil­ters both can be cre­ated sep­ar­ately, each con­tain­ing their spe­cial­ized lo­gic. This mod­u­lar­ity makes it easy to test and reason about them sep­ar­ately. Ser­vices and fil­ters are eas­ily com­posed, just like func­tions do, which makes it con­veni­ent to test chains. Ser­vices and fil­ters are meant to sep­ar­ate be­ha­viour from do­main lo­gic.

As amaz­ing as Fin­agle is, there are some things one should be aware of. To cre­ate a really re­si­li­ent ap­plic­a­tion with Fin­agle one has to be an ex­pert in its in­tern­als. Many con­fig­ur­a­tion para­met­ers in­flu­ence each other, e.g. queue size and time-outs. With a prop­erly tuned setup Fin­agle is prop­erly fast and re­si­li­ent (the de­faults are good as well, mind you). As most data centres are het­ero­gen­ous in their setup, faster ma­chines are ad­ded to the pool, and other con­di­tions change, one has to keep at­ten­tion to the tun­ing con­tinu­ously in or­der to main­tain op­timal per­form­ance.

Some gen­eral ad­vice, watch out for traffic amp­li­fic­a­tion due to re­tries, keep your timeouts low so retry is use­ful, but not as low that you in­tro­duce spuri­ous timeouts.

For ex­tra points, keep ham­mer­ing your ap­plic­a­tion un­til it breaks, find out why it breaks, fix it, and re­peat.

The future

#

In ad­di­tion to this heads up we were also given a nice in­sight in the up­com­ing things for Fin­agle.

In or­der to make more in­formed de­cision, we will get a new Fail­ure type which con­tains more in­form­a­tion in­stead of ‘just’ a Throwable. In this new Failure, an ad­ded field in­dic­ates whether it is safe to retry.

There are sev­eral is­sues with the cur­rent way of fine-tuning Fin­agle, as men­tioned, you need to be an ex­pert to use all the con­fig­ur­a­tion para­met­ers prop­erly. Next to this the con­fig­ur­a­tion is static and doesn’t take into ac­count chan­ging en­vir­on­ments and be­ha­viour of down­stream ser­vices. Be­cause the tun­ing of the para­met­ers is tightly coupled with the im­ple­ment­a­tion of Fin­agle it is also hard to change the im­ple­ment­a­tion sig­ni­fic­antly without sig­ni­fic­ant re-tuning.

In or­der to battle the last two points, Fin­agle will in­tro­duce Ser­vice Level Ob­ject­ives (SLO). The SLO is a higher-level goal that Fin­agle should strive to reach in­stead of low-level hard­coded para­met­ers. What these SLO will be ex­actly is not yet known.

The community

#

The Fin­agle team will syn­chron­ize the in­ternal Fin­agle re­pos­it­ory with the Git­hub re­pos­it­ory every Monday. They will strive to pub­lish a snap­shot ver­sion of the change as well.

For someone look­ing to write his own pro­tocol to con­nect to his ser­vice, finagle-serial is a nice pro­ject to start with. It is small enough to grasp within a day but big enough to be non-trivial.

It was found that the ParGCCardsPerStrideChunk garbage col­lec­tion op­tion, avail­able from 7u40, can halve GC times on large heaps. It is re­com­men­ded to try this para­meter. Tun­ing seems to be hard to do and is gen­er­ally done by copy­ing a ‘known good set’ of para­met­ers.

Scrooge is a good util­ity to use for Thrift and Scala as it is aware of Scala fea­tures such as Traits and Ob­jects and can gen­er­ate rel­ev­ant trans­form­a­tions for them.

When you want to con­nect to mul­tiple data-centres from a single data-centre one can use LatencyCompensation to in­clude latency times.