Akka.NET remote deployment with F#

Today I want to present how Akka.NET can be used to leverage distributed computing and discuss some of the changes in the latest F# package (Akka.FSharp 0.7.1). If you are not interested in explanation of how Akka.NET internals actually allow us to create distributed actor-based application, just skip the next part and dive directly into examples.

How Akka.NET remote deployment works

When a new actor should be created, it’s actor system deployer have to figure out if deployment should occur on local machine or on the remote node. Second option requires a network connection to be established between local/remote nodes participating in the communication. To be able to do so, each participating node needs to know a localization of the others. In Akka, this is done by using actor paths.

Actor path allows us to identify actors in scope of their systems actor hierarhies as well as localize them over the network. There are two formats of actor paths, both of them uses standard URI convention:

Local paths (eg. akka://local-system/user/parent/child) may be used to identify actors deployed (remotely or not) by our local actor system, we are referring to. In provided example actor path refers to actor with name child, which parent/supervisor actor is called parent, which resides under user guardian (which is a specialized actor supervisor existing inside actor system kernel). All of them exists in actor system named local-system.
Remote paths (eg. akka.tcp://remote-system@localhost:9001/) are similar to local ones, but a few differences occur. First we need to rename akka protocol to either akka.tcp or akka.udp to show what kind of underlying network connection we want to target. Second we have to suffix actor system name with it’s localization. This is done by providing address and port, remote node is expected to listening on.

Image below shows how actually actors refer to each other in remote deployment scenario.

When an actor is deployed remotely, a remote node is responsible for creating it’s instances, but we still refer to it using it’s actor reference and our local system context. Local/remote coordination is done by remote daemon – specialized system actor, which resides directly under /remote path of the system root. With example above we may refer to remotely deployed child using akka.tcp://sys@A:2552/user/parent/child address, while it’s true hierarchy lays under akka.tcp://sys@B:2552/remote/sys@A:2552/user/parent/child path. This allows us to preserve location transparency, hiding true underlying actor. References returned this way all almost indistinguishable from their local counterparts, allowing use of features such as monitoring, supervising and so on.

Remote deployment

Hocon configuration

To start playing with remote deployment, we should configure both endpoints to be able to maintain network communication. This is achievable through the HOCON configuration, which is the default format for both JVM and .NET Akka implementations. While you may be a little irritated by need of learning new format only to use Akka, remember:

It’s actually very easy to learn
It’s not XML ;)

Before continue, I encourage you to get yourself familiar with it.

To enable remote deployment feature, this would be a minimal Akka.NET configuration setup:

akka {  
    actor {
        provider = "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
    }
    remote.helios.tcp {
        port = 9001
        hostname = localhost
    }
}

First, every configuration node resides under common root called akka. Since remoting is not default feature of Akka.NET, we have to inform our actor system how to couple remotely deployed actors with our local actor system. This setup is done through RemoteActorRefProvider defined under akka.actor.provider config node. It will allow us to associate references to spawned actors on both local and remote systems.

Next one is remote node which defines configuration specific to remoting feature. There, we need to inform how our system will communicate with others. At the present moment it’s achievable using Helios server by default, which is lightweight and highly-concurrent TCP/UDP server developed by Aaron Stannard, one of the Akka.NET core developers – don’t confuse it with Microsoft Helios, which is totally different project. Akka.Remote will use it automatically under the hood – the only thing we have to define is address and port, under which it will listen for incoming messages. If you want port to be resolved automatically, just set 0 as the port number.

Excluding differences in system nodes localization, this configuration string may be shared by both local and remote node. You may use this configuration simply by parsing it using Configuration.parse function.

Let’s get to the point

Unlike the C#, F# API is able to construct actors using expressions compiled directly at the runtime (while still providing type safety, F# programmers are used to have). It’s achievable by leveraging F# quotations, which can be serialized and compiled on demand on other machine.

For the sample we don’t need to have two separate machines, instead we may mock them by running two Akka applications. What is necessary for the most basic example, is that both of them have to share at least Akka.FSharp and Akka.Remote assemblies as well as all of their dependencies.

install-package Akka.FSharp -version 0.7.1
install-package Akka.Remote -version 0.7.1

Remote node

The only job of the remote system is to listen for incoming actor deployment requests and execute them. Therefore implementation is very simplistic:

let config =
    Configuration.parse
        @"akka {
            actor.provider = ""Akka.Remote.RemoteActorRefProvider, Akka.Remote""
            remote.helios.tcp {
                hostname = localhost
                port = 9001
            }
        }"

[<EntryPoint>]
let main args =
    use system = System.create "remote-system" config
    System.Console.ReadLine()
    0

After running it our remote system should be listening on localhost on port 9001 and be ready to instantiate remotely deployed actors.

Local actor system instance

Second application will be used for defining actors and sending deployment requests to remote node. To do so it has to know it’s address.

To deploy our actors remotely, lets build some helper functions. To begin with, write some logic to inform our local system, that deployment should occur on the remote machine. Remember that we need to provide full address to the remote system including it’s network localization and protocol type used for communication.

open Akka.Actor
// return Deploy instance able to operate in remote scope
let deployRemotely address = Deploy(RemoteScope (Address.Parse address))

Remote deployment in Akka F# is done through spawne function and it requires deployed code to be wrapped into F# quotation.

let spawnRemote systemOrContext remoteSystemAddress actorName expr =
    spawne systemOrContext actorName expr [SpawnOption.Deploy (deployRemotely remoteSystemAddress)]

Quotations give us a few nice features, but also have some limitations:

Unlike the C# approach, here we don’t define actors in shared libraries, which have to be bound to both endpoints. Actor logic is compiled in the runtime, while remote actor system is operating. That means, there is no need to stop your remote nodes to reload shared actor assemblies when updated.
Code embedded inside quotation must use only functions, types and variables known to both endpoints. There are limited ways to define functions inside quotation expression (and no way to define types), but generally speaking in most cases it’s better to define them in separate library and share between nodes.

Last line of the spawne function is list of options used to configure actor. We used SpawnOption.Deploy to specify what deployment specifics are meant to occur. Other options may describe specifics such as message mailboxes, actor routers or failure handling strategies.

Because Akka actor system is required to negotiate deployment specifics with external nodes, it’s local instance has to be provided even thou we want to deploy our actors on the remote machines.

Finally when everything is set, we can run our example (remember that remote node must be up and running):

let system = System.create "local-system" config
let aref =
    spawnRemote system "akka.tcp://remote-system@localhost:9001/" "hello"
       // actorOf wraps custom handling function with message receiver logic
       <@ actorOf (fun msg -> printfn "received '%s'" msg) @>

// send example message to remotely deployed actor
aref <! "Hello world"

// thanks to location transparency, we can select 
// remote actors as if they where existing on local node
let sref = select "akka://local-system/user/hello" system
sref <! "Hello again"

// we can still create actors in local system context
let lref = spawn system "local" (actorOf (fun msg -> printfn "local '%s'" msg))
// this message should be printed in local application console
lref <! "Hello locally"

As a result, you should receive two messages printed in remote application console and one locally. See?

Final thoughts

Remember that Akka.NET is still in beta and all of the F# API functions are subject to change. If you have some concepts or interesting ideas, or want to help and become part of the family, you may share them on Akka.NET discussion group or directly on Github.

Appendix A: set your configuration string in application config file

While you may define configuration strings in code, the better idea is to actually store them in .config files. To be able to do so, we must simply extend configuration file with custom Akka config section:

<configSections>
    <section name="akka" type="Akka.Configuration.Hocon.AkkaConfigurationSection, Akka" />
</configSections>

Next, we can embed our Hocon-formated configuration string directly into configuration file by using <![CDATA[]]> marker (directly under main <configuration> root node):

<akka>
    <hocon>
        <![CDATA[
          ... paste your config string here
        ]]>
    </hocon>
</akka>

To load the configuration, simply call Akka.Configuration.ConfigurationFactory.Load() method.

Appendix B: Share-nothing

Since keeping actual assemblies in sync over all of the remote nodes may feel cumbersome for some people, I decided to show a little trick, which may be used to replace some of the complex data structures in your F# code. Lets look how data structure problem may be simply solved in Akka predecessor, Erlang:

go() ->
  Pid = spawn(RemoteNode, loop),
  Pid ! {hello, "world"},
  Pid ! {hi}.

loop() ->
  receive
      {hello, Msg} ->
          io:fwrite("hello ~p~n", [Msg]),
          loop();
      {hi} ->
          io:fwrite("hi~n"),
          loop()
  end.

I think, that based on your knowledge about functional programming, most of it should be at least familiar if not self-explanatory. Crux is the {hello, Msg} and {hi} syntax – these are actually instances of tuples (nested into pattern matched message receiver construct). Standard Erlang convention precises them as so called tagged tuples, which first argument is an atom – also known as symbol in other languages. Since Erlang is a dynamic language, this way we may differentiate tuples of the same size from each other.

Example below shows how usage of the Erlang’s tagged tuples could be moved into F#. The big difference is that F# has no Erlang atom equivalent. Therefore they has been replaced by F# literal integers, which gives us an advantage of human readable tags, while still counted as integers inside quotation expressions.

[<Literal>]let Hello = 1
[<Literal>]let Hi = 2

let pid = spawne system "remote" (<@ fun mailbox ->
        let rec loop() : Cont<obj, obj> = actor {
            let! msg = mailbox.Receive()
            match msg with
            | (Hello, str) -> printfn "hello %A" str
            | (Hi) -> print "hi\n"
            | _ -> mailbox.Unhandled()
            return! loop()
        }
        loop() @>) [SpawnOption.Deploy Deploy(RemoteScope(Address.Parse remoteAddr))]

pid <! (Hello, "world")
pid <! (Hi)