In my previous posts (see Akka Clustering and Remoting: The Experiment and Akka Clustering and Remoting: Scaling), I defined a distributed ping-pong application and attempted to scale that application into a cloud based Akka cluster. Thus far, deployment has been performed using JClouds and provisioning using Chef.
Now, all major cloud vendors supply a RESTful interface to their cloud infrastructures. So, at least naively, deploying compute nodes should be nothing more than translating between differing vendor REST APIs? In other words, can I build an adapter that abstracts over all these different vendor APIs? Fortunately, δ-cloud has already done most of this work for us, so much so that all I have to do is to use it!
Instead of trying to be completely general during deployment (we'll leave all that complexity to other frameworks), let us introduce a number of simplifying assumptions:
user-datasection to bootstrap our provisioning process.
By using the
user-data section to bootstrap our provisioning process, we can avoid polling or scheduled SSH loops which are present in more general frameworks such as JClouds. This however is at the small cost of needing to ensure that we use images with cloud-init or cloudbase-init installed.
Now, how might we get a compute node to inform us that it has been successfully deployed and provisioned? We could install some form of agent and get it to send a success message back to the provisioning client. However, as we are building an Akka cluster, we could also provision compute nodes so that they are all members of the same Akka cluster. That way we get the actor system to do this work for us! Clusters are often network heavy with their gossip protocols, so we'll probably need to alter a number of cluster settings (e.g.
akka.cluster.gossip-interval) in order to reduce this network loading.
As a result, this post will be about how we can build a Spray REST client for deploying cloud compute instances in a vendor agnostic manner. In addition, I wish to do this so that (at least during development and testing), I can target local virtual machines and then (as we switch or promote to production) I can retarget our application into a vendor's cloud. I will achieve this by updating and expanding Michal Fojtik's
In order to install δ-cloud, follow these installation instructions. Note that you should be using Ruby 1.9.3 and you are best installing δ-cloud from its git repository (e.g. currently, the master version of δ-cloud has networking support).
Next, create and edit the file
~/.deltacloud/config to contain your cloud vendor credentials (see
resources/deltacloud.conf for an example).
Finally, we can launch δ-cloud with a given vendor's cloud driver as follows. (Note, you can list all the available drivers with
# Here I wire up the Amazon EC2 driver to the delta-cloud (classic) front end ./bin/deltacloudd -i ec2 -f deltacloud -c ~/.deltacloud/config
Now browse to
http://localhost:3001/api to explore and use the web frontend. The documentation on the δ-cloud REST API can be found at
Performing development and testing cycles in cloud environments can be costly (in terms of per hour usage) and time consuming. So, to reduce costs and shorten the otherwise lengthy development cycles, I generally use Virtualbox virtual machines (c.f. a poor man's OpenStack cloud). In order to support this type of workflow, we will need to ensure that an appropriate δ-cloud driver for Virtualbox has been installed.
Around 2010, Michal Fojtik wrote a prototype
deltacloud-virtualbox-driver, which has otherwise not been updated. Fortunately, updating this code ends up being relatively straightforward (side note: the Virtualbox driver has to essentially equate δ-cloud instances and images).
user-data section to bootstrap our deployed instances presents one additional complication for Virtualbox instances: we will need to build an ISO containing this data so that our deployed instance (via cloud-init) may use a
NoCloud data source to start the Chef provisioning process. Fortunately, and assuming that suitable command line tools are installed (e.g.
hdiutil on OS X and
genisoimage on Linux), this can all be handled within the Virtualbox driver.
Installing the Virtualbox driver is a simple copy of the code into the δ-cloud project root:
# I assume that: in the deltacloud project root; and $EXPT3 points to the experiment-3 branch of my Github repository mkdir -p deltacloud/server/config/drivers cp $EXPT3/deltacloud/server/config/drivers/virtualbox.yaml deltacloud/server/config/drivers mkdir -p deltacloud/server/lib/deltacloud/drivers/virtualbox cp $EXPT3/deltacloud/server/lib/deltacloud/drivers/virtualbox/virtualbox_driver.rb deltacloud/server/lib/deltacloud/drivers/virtualbox
Next, we need to ensure that a suitably prepared image has been installed in Virtualbox. Ideally, the image we use should closely match those located within our target cloud infrastructures. Here, I'll use an image downloaded from Ubuntu cloud images - specifically, limited experimentation shows that we need to use trusty (i.e. 14.04) or latter.
Preparing your own image is simply a matter of ensuring that cloud-init has been installed and that
/etc/default/grub has been edited so that the variable
ds=nocloud (this is used by cloud-init to trigger the use of a NoCloud data source).
Finally, using a
deltacloud -i virtualbox invocation allows the deployment of Virtualbox machines (e.g. try this out via the δ-cloud web interface!). When we are ready to promote our code from development/testing to production, we only need to change our δ-cloud driver (plus a small change in
application.conf). Moreover, with a carefully designed application, this should be the only change (modulo configuration) that we should need to make in order to promote that code.
For ease of exposition, I do not present the δ-cloud boilerplate implementation of the Spray REST client. Instead I want to focus on how we use this REST client to deploy and provision nodes in some vendor's cloud. However, readers interested in building RESTful XML clients (in Spray) might be interested in reviewing the code?
Before I can proceed, I first need to ensure that the experiment-3 branch is checked out and built:
# Here I assume that the deltacloud Virtualbox driver is already listening at http://localhost:3001 # Checkout the experiment-3 branch git checkout experiment-3 git submodule init git submodule update # Build the cluster Debian package sbt clean sbt stage sbt debian:packageBin cp target/cluster-*.deb cookbook/cluster/files/default/ # Upload our cookbooks using knife to an Enterprise Chef server knife cookbook upload apt
knife cookbook upload hostsfile knife cookbook upload java knife cookbook upload cluster # Launch our command line console sbt console
Now, lets build clusters!
val client = ClientNode(Set("red")) // ...once the 'red' worker has joined the cluser, lets add another worker... client.provisionNode("blue") // ...whoops, once the 'blue' worker is provisioned and attempts to join the cluster, the client gets 'confused'!
So, why is the client getting 'confused' as our blue worker attempts to join the cluster? A quick review of the client console logging shows that we are receiving messages from both the red and the blue actor. However, these messages are both originating (from the client node's point of view) from the same IP address and port (i.e. socket
127.0.0.1:2552). Hence, the client is apparently unable to distinguish between each of our actors? A quick review of the worker node IP addresses (you can view these via the δ-cloud web front end) shows that both actors are provisioned on nodes that are assigned distinct IP addresses. Presumably then we are dealing with a NAT related issue here?
A more careful review of the red and blue
upstart/cluster.log files shows that in fact both actors have bound to the socket
127.0.0.1:2552 - i.e. this is not a NAT issue at all! In fact the problem here is that we have not set a hostname on our worker nodes and so, when
InetAddress.getLocalHost.getHostName is ran (to bind the actor),
localhost is being chosen as the (default) hostname. The solution here is to modify our
cloud-config script and to specify a hostname.
However, in general cloud environments, it is also necessary to configure
/etc/hosts so that the hostname maps to the NICs IP address (that we wish our actors to bind with). Here I shall handle this within the cluster recipe.
Now, with the hostname fix in place, we can at last deploy Virtualbox compute nodes and provision our ping-pong application across them!
We have seen how to use δ-cloud to deploy and provision our ping-pong application within a Virtualbox environment. In addition, I've shown how easy it is to promote our application to a general cloud vendor such as Amazon or Rackspace. However, as with the Akka Clustering and Remoting: Scaling post, actually deploying our application into a vendor cloud will currently fail. In the next post, I will look at how these networking issues might be resolved and how we can finally play ping-pong between a local NATed machine and a cluster located in a vendor cloud.