<img height="1" width="1" src="https://www.facebook.com/tr?id=1076094119157733&amp;ev=PageView &amp;noscript=1">

Serf's Single Configuration File

Posted by Charles Rice on Wed, Aug 31, 2016

Serf is a fantastic lightweight technology that can help to orchestrate your infrastructure in the cloud such as dynamically updating a load balancer or backing up a database based on events, even within your Docker containers. Once you're finished playing with it and creating a proof of concept the time comes to integrate it and start automating that integration. This is where the trouble begins. Whilst the documentation exists on the Serfdom official pages it is perhaps too succinct and only provides two samples, neither of which was suitable for us to get going. This blog post aims to remedy this and provide you with some insight into: creating a single config file, adding a handler, hooking it up to your Serf agent, the seeding procedure we use and debugging to get your started.

Creating the Config File

Before creating the config file we need to know how to give a config file to the Serf agent. The easiest way to do this is via command line option:


serf agent -config-file=/etc/serf/myconfig.json
As you may have noticed above, the config file is a json file. This makes it very easy to manipulate and create since it's widely implemented and very human-readable. Below is the bare minimum set of options required for the config file.


{
    "node_name": "ip-10-30-40-50",
    "protocol": 4,
    "advertise": "10.30.40.50",
    "bind": "0.0.0.0",
    "encrypt_key": "",
    "rpc_addr": "0.0.0.0:7373",
    "log_level": "info"
}

Some of you may have noticed from the node_name that this example is from running in an AWS infrastructure, but the same rules still apply when running locally or on other platforms. Purposes of each of these options are as follows:

  • node_name: Name of your node, usually the host name for simplicity.
  • protocol: Serf protocol version; non-essential but useful when you need to upgrade a long running cluster.
  • advertise: The IP address other Serf nodes should contact you with.
  • bind: We found that binding to 0.0.0.0 (all interfaces) was the best approach for portability. Default port is 7946.
  • encrypt_key: Enables secure communications, can be left blank. Must be 16-bytes and base64 encoded. Recommended to use serf keygen.
  • rpc_addr: Allows programmatic control of this Serf node, also used by the supplied Serf Cli.
  • log_level: Self-explanatory - info is sufficient, debug giving more details and trace supplying the most.

Adding a Handler

Now that we have a running configuration we should use Serf for what it was built for - reacting to events that occur throughout the cluster. Handlers are Serf's method of giving you an entry point to then react to events such as other members joining, leaving or failing. You would implement it like this:


"event_handlers" : [
  "/path/to/my/script.sh"
]
As you can infer from above, the path it is installed to doesn't actually matter although it is better to follow best practices and put it within the /etc/serf/handlers folder. If desired you could specify multiple handler files if you split functionality into several shell scripts for example. We prefer to use Python for this as it offers great readability and allows us to compose a more structured program, but you could easily use other langauages too. Below is what the config file would look like with our new Serf handler:


{
    "node_name": "ip-10-30-40-50",
    "protocol": 4,
    "advertise": "10.30.40.50",
    "bind": "0.0.0.0",
    "encrypt_key": "",
    "event_handlers" : [
        "/etc/serf/handlers/serf_handler.py"
    ],
    "rpc_addr": "0.0.0.0:7373",
    "log_level": "info"
}

Seeding your Cluster

Now that we have a Serf agent that is setup correctly and has a handler reacting to events, the next step is to actually form a cluster. Borrowing some seeding concepts from Datastax Cassandra we setup three Serf agents to act as "seeds" and registered them to DNS names for simplicity. Using a DNS service will help significantly with this step. Simply add a start_join attribute like this:


"start_join": [
  "www.serf-seed-1.com",
  "www.serf-seed-2.com",
  "www.serf-seed-3.com"
]

Putting it all together

Taking all of the elements explained previous we can put together a configuration file that has the minimum of what we need, is easy to manipulate programmatically and looks something like the following:


{
    "node_name": "ip-10-30-40-50",
    "protocol": 4,
    "advertise": "10.30.40.50",
    "bind": "0.0.0.0",
    "encrypt_key": "",
    "event_handlers" : [
        "/etc/serf/handlers/serf_handler.py"
    ],
    "tags": {
        "role": "database"
    },
    "rpc_addr": "0.0.0.0:7373",
    "log_level": "info",
    "start_join": [
        "www.serf-seed-1.com",
        "www.serf-seed-2.com",
        "www.serf-seed-3.com"
    ]
}

Debugging

Now that you've been given some insight in how to use a single file to setup your Serf agents I will finish off by giving you some concise handy hints to debug and troublshoot should you run into any issues attempting the above.

Handler execution errors

What this means is there's a problem with one of three things: the program to run the script cannot be found, the script itself cannot be found or there's an issue with the code in your script. The following may help:

  • Ensure you specify the right extension or program at the top of your script. e.g. Name it myscript.py and add #!/usr/bin/env python at the top.
  • Specify the full path of your event handler and make sure to use double quotes. Double check your script is actually there!
  • I can't help you with this, test your script in isolation by running /path/to/your/script.sh and verify it works.

Unable to join cluster

This indicates a problem reaching your seed nodes. You can try using direct IP addresses and checking that the hostnames you've used are reachable by running host www.serf-seed-1.com in your favourite terminal.

Summary

In this post we walked through the basics of a Serf configuration file, why they're needed and how to add the essentials such as seed nodes to bootstrap a cluster and a handler to perform your orchestration code. Finally I listed some common problems you may encounter attempting to setup your own Serf cluster and some diagnosis tips. This is only the tip of the iceberg and you can create more sophisticated setups that involves handlers for specific types of messages and enhancing your handlers to make use of the different tags.

Best of luck with your setups - until next time, Chuck.

Posts by Topic

see all

Subscribe to Email Updates