<img height="1" width="1" src="https://www.facebook.com/tr?id=1076094119157733&amp;ev=PageView &amp;noscript=1">

Akka Clustering and Remoting: Alternative Deployment Strategy

Posted by Carl Pulley on Tue, Jun 24, 2014

Need help on your next project?

Introduction

In my previous posts (see Akka Clustering and Remoting: The Experiment and Akka Clustering and Remoting: Scaling), I defined a distributed ping-pong application and attempted to scale that application into a cloud based Akka cluster. Thus far, deployment has been performed using JClouds and provisioning using Chef.

Now, all major cloud vendors supply a RESTful interface to their cloud infrastructures. So, at least naively, deploying compute nodes should be nothing more than translating between differing vendor REST APIs? In other words, can I build an adapter that abstracts over all these different vendor APIs? Fortunately, δ-cloud has already done most of this work for us, so much so that all I have to do is to use it!

Instead of trying to be completely general during deployment (we'll leave all that complexity to other frameworks), let us introduce a number of simplifying assumptions:

  • deployments, at any point in time, will always be to a single vendor cloud
  • we will use the user-data section to bootstrap our provisioning process.

By using the user-data section to bootstrap our provisioning process, we can avoid polling or scheduled SSH loops which are present in more general frameworks such as JClouds. This however is at the small cost of needing to ensure that we use images with cloud-init or cloudbase-init installed.

Now, how might we get a compute node to inform us that it has been successfully deployed and provisioned? We could install some form of agent and get it to send a success message back to the provisioning client. However, as we are building an Akka cluster, we could also provision compute nodes so that they are all members of the same Akka cluster. That way we get the actor system to do this work for us! Clusters are often network heavy with their gossip protocols, so we'll probably need to alter a number of cluster settings (e.g. akka.cluster.gossip-interval) in order to reduce this network loading.

As a result, this post will be about how we can build a Spray REST client for deploying cloud compute instances in a vendor agnostic manner. In addition, I wish to do this so that (at least during development and testing), I can target local virtual machines and then (as we switch or promote to production) I can retarget our application into a vendor's cloud. I will achieve this by updating and expanding Michal Fojtik's deltacloud-virtualbox-driver code.

δ-cloud

In order to install δ-cloud, follow these installation instructions. Note that you should be using Ruby 1.9.3 and you are best installing δ-cloud from its git repository (e.g. currently, the master version of δ-cloud has networking support).

Next, create and edit the file ~/.deltacloud/config to contain your cloud vendor credentials (see resources/deltacloud.conf for an example).

Finally, we can launch δ-cloud with a given vendor's cloud driver as follows. (Note, you can list all the available drivers with deltacloudd -l):

# Here I wire up the Amazon EC2 driver to the delta-cloud (classic) front end
./bin/deltacloudd -i ec2 -f deltacloud -c ~/.deltacloud/config

Now browse to http://localhost:3001/api to explore and use the web frontend. The documentation on the δ-cloud REST API can be found at http://localhost:3001/api/docs.

Notes:

  • If you do use the web frontend to control the δ-cloud resources, then please do note that δ-cloud doesn't necessarily prompt for confirmation or perform sufficient error checking before executing actions!
  • For those of you interested in alternative Web UIs that leverage the δ-cloud API, you might also be interested in looking at CloudManage?

Provisioning in Development and Testing Environments

DeltaCloud

Performing development and testing cycles in cloud environments can be costly (in terms of per hour usage) and time consuming. So, to reduce costs and shorten the otherwise lengthy development cycles, I generally use Virtualbox virtual machines (c.f. a poor man's OpenStack cloud). In order to support this type of workflow, we will need to ensure that an appropriate δ-cloud driver for Virtualbox has been installed.

Around 2010, Michal Fojtik wrote a prototype deltacloud-virtualbox-driver, which has otherwise not been updated. Fortunately, updating this code ends up being relatively straightforward (side note: the Virtualbox driver has to essentially equate δ-cloud instances and images).

Using the user-data section to bootstrap our deployed instances presents one additional complication for Virtualbox instances: we will need to build an ISO containing this data so that our deployed instance (via cloud-init) may use a NoCloud data source to start the Chef provisioning process. Fortunately, and assuming that suitable command line tools are installed (e.g. hdiutil on OS X and genisoimage on Linux), this can all be handled within the Virtualbox driver.

Installing the Virtualbox driver is a simple copy of the code into the δ-cloud project root:
# I assume that: in the deltacloud project root; and $EXPT3 points to the experiment-3 branch of my Github repository
mkdir -p deltacloud/server/config/drivers
cp $EXPT3/deltacloud/server/config/drivers/virtualbox.yaml deltacloud/server/config/drivers
mkdir -p deltacloud/server/lib/deltacloud/drivers/virtualbox
cp $EXPT3/deltacloud/server/lib/deltacloud/drivers/virtualbox/virtualbox_driver.rb deltacloud/server/lib/deltacloud/drivers/virtualbox

Next, we need to ensure that a suitably prepared image has been installed in Virtualbox. Ideally, the image we use should closely match those located within our target cloud infrastructures. Here, I'll use an image downloaded from Ubuntu cloud images - specifically, limited experimentation shows that we need to use trusty (i.e. 14.04) or latter.

Preparing your own image is simply a matter of ensuring that cloud-init has been installed and that /etc/default/grub has been edited so that the variable GRUB_CMDLINE_LINUX_DEFAULT contains ds=nocloud (this is used by cloud-init to trigger the use of a NoCloud data source).

Finally, using a deltacloud -i virtualbox invocation allows the deployment of Virtualbox machines (e.g. try this out via the δ-cloud web interface!). When we are ready to promote our code from development/testing to production, we only need to change our δ-cloud driver (plus a small change in application.conf). Moreover, with a carefully designed application, this should be the only change (modulo configuration) that we should need to make in order to promote that code.

Reactive Provisioning using Spray, Akka and Chef

For ease of exposition, I do not present the δ-cloud boilerplate implementation of the Spray REST client. Instead I want to focus on how we use this REST client to deploy and provision nodes in some vendor's cloud. However, readers interested in building RESTful XML clients (in Spray) might be interested in reviewing the code?

Before I can proceed, I first need to ensure that the experiment-3 branch is checked out and built:

# Here I assume that the deltacloud Virtualbox driver is already listening at http://localhost:3001
# Checkout the experiment-3 branch
git checkout experiment-3
git submodule init
git submodule update
# Build the cluster Debian package
sbt clean
sbt stage
sbt debian:packageBin
cp  target/cluster-*.deb cookbook/cluster/files/default/
# Upload our cookbooks using knife to an Enterprise Chef server
knife cookbook upload apt
knife cookbook upload hostsfile knife cookbook upload java knife cookbook upload cluster # Launch our command line console sbt console

Now, lets build clusters!

val client = ClientNode(Set("red"))
// ...once the 'red' worker has joined the cluser, lets add another worker...
client.provisionNode("blue")
// ...whoops, once the 'blue' worker is provisioned and attempts to join the cluster, the client gets 'confused'!

So, why is the client getting 'confused' as our blue worker attempts to join the cluster? A quick review of the client console logging shows that we are receiving messages from both the red and the blue actor. However, these messages are both originating (from the client node's point of view) from the same IP address and port (i.e. socket 127.0.0.1:2552). Hence, the client is apparently unable to distinguish between each of our actors? A quick review of the worker node IP addresses (you can view these via the δ-cloud web front end) shows that both actors are provisioned on nodes that are assigned distinct IP addresses. Presumably then we are dealing with a NAT related issue here?

Akka-NATing

A more careful review of the red and blue upstart/cluster.log files shows that in fact both actors have bound to the socket 127.0.0.1:2552 - i.e. this is not a NAT issue at all! In fact the problem here is that we have not set a hostname on our worker nodes and so, when InetAddress.getLocalHost.getHostName is ran (to bind the actor), localhost is being chosen as the (default) hostname. The solution here is to modify our cloud-config script and to specify a hostname.

However, in general cloud environments, it is also necessary to configure /etc/hosts so that the hostname maps to the NICs IP address (that we wish our actors to bind with). Here I shall handle this within the cluster recipe.

Now, with the hostname fix in place, we can at last deploy Virtualbox compute nodes and provision our ping-pong application across them!

Conclusion

We have seen how to use δ-cloud to deploy and provision our ping-pong application within a Virtualbox environment. In addition, I've shown how easy it is to promote our application to a general cloud vendor such as Amazon or Rackspace. However, as with the Akka Clustering and Remoting: Scaling post, actually deploying our application into a vendor cloud will currently fail. In the next post, I will look at how these networking issues might be resolved and how we can finally play ping-pong between a local NATed machine and a cluster located in a vendor cloud.

 

 Need help on your next project?

Recent Posts

Posts by Topic

see all

Subscribe to Email Updates