Naive agent-based backends

Seeing the popularity of elmish architecture and attempts at implementations of backends based on it (or any unsupervised cooperative actor loops) I thought I’d share my thoughts.

Desirable characteristics

Most naive implementations of synchronous (HTTP/RPC) APIs are perfectly content to return some equivalent of 500 code at a drop of a hat. My DB timed out? 500! My message broker node has crashed and I need to reconnect? 500! Etc. etc. You get the idea.

But this and other kinds of recoverable problems happen all the time in distributed computing and scenarios with unpredictable load. Remember, network is unreliable, as is your cloud VM and the physical machine under it. If your service/agent is on the edge, the constrained device might be under load doing whatever its primary job.

Here’s another consideration: What if it’s in the middle of a chain of APIs, as is common with microservices? Wether you return an error or not, what is the recovery strategy for the earlier nodes in the chain? Retry? Then something like this should look familiar:heavy-mallet


What about timeout constraints higher up the call chain? At some point, you’d end up  returning the error all the way up the call chain and not only this is an overhead you could have avoided, now it’s your user’s problem.

Conversely, is it ok to just drop the error? This is what unsupervised actors and elmish dispatch loop do. In elmish at least the API tries to prepare you for dealing with the possibility of errors, but if something does slip through, the best we can do is log it.

And logging it is an example of data loss. There’s no longer the context of the call, so there may or may not be a data that was passed in required to resolve the problem even if you found out about it at later date.  This might be ok in the UI, but on the backend/edge it could lead to hard to diagnose and solve business problems.

So, one desirable characteristic that’s different for backends and edge nodes is resilience. If we can recover from an error, we should do so without making it our client’s responsibility.

Achieving resilience

Yes, I’m going to talk about store-and-forward architecture, which may seem ironic, considering the picture above comes from reddit – one of the most prominent users of RabbitMQ, but the fact is reddit scales far better than most websites could.

The essential ingredients of resilience are asynchronous APIs (messaging) and ack/nack functionality of the messaging infrastructure.

Asynchronous messaging allows us to change the problem of timing from “It’s time-sensitive” to “It’s time-relevant” – we just have to process the messages in certain order, not on a strict timeline.

Ack/nack allows us to guarantee that the message is processed completely, which is what makes asynchronous API trustworthy – we’ll never loose the message or the fact that there might have been a problem processing it.

Ack/nack is what makes it “fire-and-forget” – often misinterpreted analogy. Imagine if military fired the rockets thinking “wether it hit the target or not, we can forget about it once it’s fired”. No, of course we care what happens, and it’s because there are recovery mechanisms in place that we can kinda forget about it, which logging and unsupervised actors don’t provide.

Speaking of actors

If you read the original papers on actors, they were meant to be a reasoning tool in the face of high processing complexity. The original constraints got watered down with the ability to address any actors in the cluster and the approach has become a tactical tool for relatively safe cooperative concurrency. However…

Your single actors node can process $300K/s messages? I’m not impressed, the number is meaningless not only because your problem doesn’t translate into mine, but also because your actors are unsupervised. Show me the supervised numbers, where individual message can be shown to be fully processed, replayed (wether successfully or as part of the compensation workflow while handling a non-recoverable error), and while we are at it – can it throttle the sources of events so that my downstream processing is not overwhelmed?

Higher-level abstractions

That brings me to “streaming” – modern day abstractions that define concepts of sources, data streams and sinks. But it doesn’t stop there, if you take a look at Apache Beam, you’ll see a standard defining not only these essentials, but higher-level operators like grouping, windowing and many others. The standard is implemented by several different data processing frameworks, so if we are going to build something resilient let’s focus on stream processing.

Edit: Conclusion

Elmish abstractions lack the way to implement Ack/Nack, define a data stream, as well as any means of facilitating update parallelism. If none of that matters , for example – in a locally hosted node.js backend to support user interaction, then you are OK. For a more robust implementation look elsewhere… maybe check out FsShelter which I’ll be presenting at OpenFSharp in September.

Naive agent-based backends

Elmish now supports RemoteDev time-travelling debugger

Screen Shot 2017-01-11 at 11.33.13 AM.png

Since day 1 elmish, developed at Prolucid for our front-end applications supported console logging of states and updates taking place in the application. You have to see it to appreciate how powerful of a feature that is and for a while it was enough. That is until ever-prolific @forki decided he wants a full blown Elm-like debugger 🙂

Thanks to RemoteDev tools and its developer Mihail we now have the experimental support in Elmish that makes the features like time-traveling and import/export available to apps targeting web and mobile!

For details see the README, the React sample and the React Native sample.

The Chrome and Firefox extensions are easy to install, but the setup for mobile is a little-bit complicated, due to the fact that the communications are done over the network. It requires either a connection to a public cluster or running a local server, with everything that entails – making sure the routes are setup, HTTP/s traffic possible, ports are forwarded, etc. But it’s all very well documented in RemoteDev server repo.

Big Thanks to Steffen for pushing for the feature and Mihail and Alfonso for their support in making it possible!

Elmish now supports RemoteDev time-travelling debugger

Authoring Fable bindings: inheritance and variance in action.

JavaScript cheats. If you want to pass some properties, there’s no need for a type to describe them – just return anything you want inside of curly braces and you are done. Of course there’s a cost, but that not what this post is about.

How does one achieve this level of freedom in order to integrate with JavaScript libraries out there when writing in a statically typed language – F#? Fable provides several mechanisms for transparent interop with the rest of JS world:

  • Record Types – going from typed to untyped is not a problem;
  • Discriminated unions with a special attribute [<KeyValueList>].

Currently a typical way to create new fable bindings for a library starts with TypeScript definitions imported via ts2fable.  The tool produces F# type definitions that closely resemble the OO roots of TypeScript definition, but are not particularly interesting to work with from a functional-first language. So the usual second step when authoring a binding is a creation of a “helper” DSL – conversion of generated interfaces to KeyValueList discriminated unions and addition of factory functions to take lists of DU instances that describe various properties of the target. Let’s take a look at an example, from React JSX intro:

Brief detour: JSX is template language that lets you mix React elements described in XML/HTML-like syntax with JS6, it’s transpiled later into plain old JS. So in lines 2-4 we see a construction of h1element with implicit child element (“hello…” text) and an explicit className and key attributes set. With regards to React, key is unnecessary and may even be invalid, but this post is not about React and the attribute is perfect to discuss the types involved.

We’ll skip the actual JS definition of the h1, just keep in mind that we are setting a couple of properties and let’s dive into TypeScript definition for each:

We can see that TypeScript introduced an interface hierarchy to describe something that would be a runtime composition in JS – object is just an array, indexed by property name. Leaving aside for now the interesting use of type intersection, let’s take a look at what F# binding we get from ts2fable:

Unsurprisingly we get a couple of interfaces Props and HTMLAttributes, unified in HTMLProp. Let us consider how we would use the interfaces like that… we’d need an object of type HTMLProp that we’d call an interface method on, but it’s only useful for callbacks. The callbacks are contravariant -as long as it receives the “biggest” type we are type-safe.

But how would we construct an object in type-safe manner using these definitions? The answer is we can’t. Constructors need a covariant list of properties.

Let’s see what the idiomatic F# DSL offers on top of the imported definitions:

Here we see the DUs representing the properties from the imported interfaces and the “factory” functions to facilitate assembly of the element objects. We also see a marker interface IProp that doesn’t exist anywhere in the previously seen hierarchy. Why do we need the marker interfaces? To answer that question, let’s see the usage:

We pass HTMLAttr.ClassName and Props.Key as the list elements and a child text element.

What is interesting is that here we have a typed list with instances coming from different types. In JS we just bundle anything we like with {} , in TypeScript we can do the same, but there are also “intersection types” that allow one to express mixing of various properties together, in F# we need the artificial marker interface to relate the unrelated types, so that our list could be constructed.

What we see taking place is transition from interface definitions usable by contravariant callbacks to a covariant list of properties that can be passed into the createElement and it required the unifying interface type to make it possible.

Unlike the fairly simple React hierarchy, which if you squint looks like a match for the original inheritance tree, ReactNative is more involving and requires a lot of unifying interfaces – one for each valid combination of properties for a given element and requires careful thought and consideration. It’s a work in progress. On the other hand what JS can only check at runtime, F# can check at compile time, as well as provide a nice IntelliSense experience!

Authoring Fable bindings: inheritance and variance in action.

Cross-platform UIs with F# and Fable

If you are small vendor and your primary focus hasn’t been in designing UIs, entering the field today presents you with too many choices. However, if you’d like to use existing expertise and develop new ones in way that would accommodate broad range of platforms: Web, Mobile, Windows and, ideally, OSX and linux – the choices shrink dramatically.

We have some WPF expertise in house, but due to premature demise of Silverlight it has become a non-transferrable skill. Xamarin seemed like a way forward, but it would fragment our investment into Mobile/Desktop and despite Xamarin’s accomplishments (and they are impressive) it remains a fragile niche and having tried it we decided to keep looking.

React (and its Native derivative) with “learn once, write anywhere” approach seems like a promising direction but it has one (big) problem – JavaScript. Having built statically-verifiable code the weak and dynamic nature of JS leaves the language entirely unattractive. On the other hand JavaScript as ecosystem is like a lab full of petri dishes, rapidly blossoming and quickly killing off an infinite stream of ideas. It’s great of course that it’s happening, but trying to figure out the minimal viable combination of tools and libs… the fatigue sets in rather quickly.

Over the past couple of years we at Prolucid have been building up our F# skills developing backend systems and looking at Elm with its beautiful implementation of “model view update” architecture and Fable with its amazing capabilities I realized we may have our way forward.

Building hardware and low-level device software I expect we’ll be dropping into native quite a bit, but doing native in the tools that were designed for it seems like an excellent idea anyway. At the same time we’ll be able to reuse tools, core logic and models across platforms and tiers.

What was lacking is F# implementation of Elm’s dispatch loop, similar to what Redux does, but w/o all the overhead seemingly designed to overcome the language shortcomings. There’s an implementation in fable-virtualdom from Tomas Jansson, but it’s tied at the moment with virtual dom management.

To that end an early build of Elmish is now available on npm. Elmish is a small library that has been designed following Elm’s concepts and terminology and it should work with React, ReactNative, virtualdom and any other DOM/rendering framework.

There are samples for React (CounterTodoMVC) and React Native.

With exception of F# syntax and explicit dispatch function being passed around one could follow Elm’s documentation when studying this approach to writing a UI, but hopefully I’ll get some time to write some docs in the next few weeks.

Many thanks to Fable contributors for the help with the ecosystem and for making F# a competitive language to write cross-platform UIs in!

Cross-platform UIs with F# and Fable

Protoshell and Thriftshell update

As part of FsShelter development I had to implement the multilang serilizers for Storm, originally building against then current Storm 0.10.0.

I’ve started with Thrift, thinking that since its already a part of Storm runtime it would make the adoption easier compared to protobuf and given similar  characteristics the downgrade in performance would not be noticeable. After some testing it turned out that Thrift performed nearly at the speed of JSON (better with some payloads, worse with others), which might require some explanation.

Unlike monolithic protobuf, Thrift has a pluggable model for pretty much everything. So when people say Thrift they should qualify at least two things: Transport and Protocol. Thirft looks comparable to protobuf only when Compact protocol is used. Compact however has a caveat, it doesn’t work with streaming transports, unless you implement custom framing logic to achieve something similar to protobuf’s ParseDelimitedFrom functionality. And Storm is all about streaming, which is why I’m deprecating the support for Thrift. Unless someone wants to maintain it I’ll be removing Thrift support from the future releases of FsShelter.

Protoshell on the other hand gets an update – Storm 1.0 has been released and some packages have been renamed. The new 1.0.1 release of Protoshell is now available and has been tested to work with latest Storm, so now FsShelter can benefit from massive performance improvements made in Storm.

FsShelter does not require a new build to benefit from this release. All one needs to start running FsShelter components against Storm 1.0.1 is the new server-side serializer implementation, which can be referenced directly from github as a paket dependency and included with the topology for deployment.

Protoshell and Thriftshell update

An easy way to try FsShelter

Thanks to docker, trying something out w/o having to figure out all the dependencies and pollute your system has become really easy. This is how I started with Storm and this is the way we now help others to try FsShelter as well – fsshelter-samples container.

The container includes an installation of Storm, Mono, F# and a pre-built clone of FsShelter repo. Original build of Mono (4.2.1) caused processes to crash now and then and was an interesting study in how Storm deals with failures and what it means for processing guarantees. Current version (4.2.3) runs solid and may deprive you from witnessing Storm restarting all the components… you may have to crash them yourself 🙂

An easy way to try FsShelter

FsShelter: a Storm shell for F#

About a year ago Prolucid adopted Apache Storm as our platform of choice for event stream processing and F# as our language of choice for all of our “cloud” development.

FsStorm was an essential part that let us iterate, scale and deliver quickly, but even from the earliest days it was obvious that the developer experience could be improved. Unfortunately, it meant a complete rewrite of FsStorm:

  • FsStorm DSL is a really thin layer on top of Nimbus API model:
    • has explicit IDs when describing components in a topology
    • uses strings in all the names
    • matching of inputs/outputs is not guaranteed
  • FsStorm uses Json AST as it’s public API:
    • messages, tuples, configuration
    • serialization mechanism is hard-baked into the API

We’ve worked around some of the problems, usually by writing more code.

It actually makes sense that Storm itself doesn’t care about the type of the tuples/fields. It runs on JVM, which is very much typed, and it relies on sub-class polymorphism to make things tick. However the public API for the tuples looks like an afterthought in every language. But we figured, there is this “compiler” that can do “type checking” for us, let’s make it work! Maybe we can even make it faster if we replace Json with Protobuf?

Coming up with the new DSL that would allow the components to consume and emit tuples of various (static) types on multiple streams was an interesting experience and led to some strange places. A lot has been written on F# DSLs, but none of that applied directly. Can I use “just functions”? Do I need a type provider? A computation expression? A compiler as a service?


After a few false starts I found the desired paradigm that could be expressed in F# succinctly. As it usually happens, once I gave up on certain notions (building “any purpose” graph from a single source in this case), the result was pretty simple. And so, after a few weeks of journey and discovery, we are releasing FsShelter: a way to program Storm with F# in a statically typed fashion.

Many thanks to Tomas Petricek, Scott Wlaschin, Andrew Cherry and Erik Tsarpalis, without them FsShelter wouldn’t have been possible.

FsShelter is currently in beta and any feedback is welcome and appreciated.

FsShelter: a Storm shell for F#