First, I installed the sbt-plugin for Jenkins from the list of available plugins under Manage Plugin.

Once the plugin installation completes and Jenkins is restarted, I verified that sbt-plugin is now available as one of the installed Jenkins plugins.

Now that sbt-plugin is available on Jenkins, I had to configure a sbt launcher jar. I downloaded the very latest sbt launcher jar [0.11.2] from here, and saved it on our Jenkins build server. In order to complete the configuration, I had to specify the name and location of the sbt launcher jar under Jenkins configuration.

Jenkins sbt plugin allows you to install multiple versions of sbt launcher jar files giving you the option of choosing the most appropriate one for your project.

All that remained now was to create a Jenkins build job which used sbt-launcher to build Spec2Spring. To do that I simply selected Build using sbt from Add build step.

Finally, I selected my preferred sbt launcher and specified the actions I wanted to run. And that was all I had to do!

-Dsbt.log.noformat=true sbt flag keeps the console output clean and tidy!

Note: To avoid frequent OutOfMemory errors, add something like the following JVM flags to your Jenkins build configuration (notice in particular the stack and perm gen size parameters):

-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx512M -Xss16M

For more detailed discussion of the Cake pattern, see Mark’s Cake pattern in depth

Before we rush into condemning one or the other, we feel that what you need to think about what kind of application you’re writing. We’ll go as far as to say that if you need JMS, JMX, scheduling, WebFlow, …, then you should stick with the Spring Framework. By all means implement the components that make up your Spring application in Scala! If these JEE dinosaurs do not come anywhere near your application, then Play 2 in Scala, using the Cake pattern might just be the perfect thing for you!

Dependency injection

Spring Framework, we have multiple modules; each module contains the code that is appropriate for the tier that the module implements. The code means the Java interfaces & their implementations as well as the Spring configuration (typically in the META-INF/spring/module-context.xml file). The types of the dependencies are checked at compile-time: if we haven’t defined the UserService interface, for example, and tried to use it in the UserController, the compilation would fail. However, the compiler cannot check that the appropriate implementation of the UserService will be available to be injected to the UserController at runtime. The application is loosely coupled, which is good, but we must supply appropriate metadata (the annotations & the Spring XML) so that Spring can construct the application.

Dependency-injection applications must provide valid metadata so that the DI framework can construct their components & satisfy the dependencies between them at runtime.

Let’s write a trivial application that contains a repository and a service. We will follow the recommended code to interfaces approach. This gives us the following code in the repository:

public interface UserRepository {
  List<User> findAll();
}

@Repository
public class ORMUserRepository implements UserRepository {
  private SessionFactory sessionFactory = // construct the SF
  public List<User> findAll() {
    // not real Hibernate code, but close enough
    return sessionFactory.getCurrentSession().query(...);
  }
}

We now combine this code with the Spring context file, which instructs the DI core to create the ORMUserRepository component.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:context="http://www.springframework.org/schema/context"
  xsi:schemaLocation="...">

  <context:annotation-config/>
  <context:component-scan base-package="...repository"/>

<beans/>

We implement the code in the services in using a similar pattern: we have the UserService interface and we implement it in some class. Crucially, we will rely on Spring to inject a valid implementation of the UserRepository to the instance of the DefaultUserService at runtime, using the configuration we provide.

public interface UserService {
  List<User> findAll();
}

@Service
public class DefaultUserService implements UserService {
  private final UserRepository userRepository;

  @Autowired
  public DefaultUserService(final UserRepository userRepository) {
    if (userRepository == null)
      throw new IllegalArgumentException("Off Santa's list!");
    this.userRepository = userRepository;
  }

  public List<User> findAll() {
    return this.userRepository.findAll();
  }
}

The services module needs its configuration, too. And so, we give the familiar /META-INF/spring/module-context.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:context="http://www.springframework.org/schema/context"
  xsi:schemaLocation="...">

  <context:annotation-config/>
  <context:component-scan base-package="...services"/>

<beans/>

We have the components and the metadata. All we need now is to let Spring Framework create all the components. In other words, the ApplicationContext implementation that the Spring Framework provides will become our application:

public class App {

  public static void main(String[] args) {
    ApplicationContext application =
      new ClassPathXmlApplicationContext(
        "classpath*:/META-INF/spring/module-context.xml"));

    UserService service = application.getBean(UserService.class);

    // use it!
    service.findAll();
  }
}

The Spring Framework became our application! All we have is a shell that boots-up Spring, as shown on Figure 1.

Figure 1. Spring DI application

The Cake Pattern

If you are writing new Scala application, you might not want to take in all the baggage that Spring Framework brings with it. (Slf4j, anyone?)

You would typically use the Cake Pattern to achieve something akin to dependency injection. Jonas Boner explained the Cake Pattern in one of his infamous Real-World Scala Series blog post. What we would try here is to use the Cake pattern to implement the same DI scenario in Scala and then compare that with its corresponding Spring implementation. In doing that we would hope to clear out some of the misty clouds over Scala and DI.

We also want to have a repository and a service, but without the XML configuration file. So, we need not only the repository and service interfaces themselves, but we also need components in which they “live”. Starting with the repository, we have:

trait UserRepositoryComponent {
  def userRepository : UserRepository

  trait UserRepository {
    def findAll: List[User]
  }
}

trait ORMUserRepositoryComponent extends UserRepositoryComponent {
  def userRepository = new ORMUserRepository(sf) //we need to actually instantiate the session factory here

  class ORMUserRepositry(val sf: SessionFactory) extends UserRepository {
    def findAll: List[User] = sf.query(...)
  }
}

Good. We have the repository tier interface the UserRepositoryComponent provides a way to get to the UserRepository instance. We have also shown an implementation of the repository interfaces: the ORMUserRepositoryComponent‘s userRepository function gives an ORM-based implementation of the UserRepository.

Now, we would like to define the interface to the services tier:

trait UserServiceComponent {
  def userService: UserService

  trait UserService {
    def findAll: List[User]
  }
}

OK, we have the repository interface & implementation and the service interface. The implementation of the service will need to have access to some implementation of the repository interface. Naturally, we don’t want to specify which concrete repository interface implementation we want to use. In fact, we would like to let the users of our code specify arbitrary implementation of the repository interfaces. Good thing we have the composition inheritance!

trait DefaultUserServiceComponent extends UserServiceComponent {
  this: UserRepositoryComponent => 

  def userService = new DefaultUserService

  class DefaultUserService extends UserService {
    def findAll = userRepository.findAll
  }
}

Notice in particular this: UserRepositoryComponent =>. This means that the instance of DefaultUserServiceComponent depends on the UserRepositoryComponent. (It’s a little more subtle than that, but for the purposes of this post it will do.) This is in function equivalent to Spring’s @Autowired public DefaultUserService(final UserRepository userRepository). Just like the Spring application we have created, the Scala code does nothing yet. We need to wire it together in some kind of application!

object Application {
  val userServiceComponent = new DefaultUserServiceComponent with ORMUserRepositoryComponent
  val userService = userServiceComponent.userService
}

val userService = Application.userService
userService.findAll

In Scala, there is no ApplicationContext, so we have to implement our own application. But where’s the dependency injection? Here: new DefaultUserServiceComponent with ORMUserRepositoryComponent, notice the with ORMUserRepositoryComponent. Figure 2 shows the Scala application.

Figure 2. Scala DI

The comparison

As you can see, the two approaches are completely different in their implementation. DI in Spring Framework is a runtime business and the component separation is left to the way in which we structure the configuration files. DI in Scala/Cake is a compile-time business with sharper component separation. Figure 3 gives the overall comparison.

Figure 3. The big picture

Summary

As we have shown here, both pattern represent in function a kind of dependency injection & management. However, the crucial difference is that the dependency injection (in Spring Framework) is useful to manage components, leaving us to carefully design the components. The Cake Pattern allows us to inject & manage functionality. We use traits to assemble components with the functionality we require. It is a different approach to design. Let me repeat:

In plain dependency injection, we create components and we assemble these components together to form an application.
Using the Cake Pattern, we create pieces of functionality and we assemble the functionality to form the application.

So, to make sure that we can run Haskell “in the cloud” (especially considering Julian Porter’s CloudHaskell work), I decided to include the Haskell runtime and Happstack framework in Cloud Foundry.

Success!

And guess, what? It worked!

Before I tell you all the details and point you to my Github fork of vcap, I’ll show you, starting with Figure 1, which shows that vmc runtimes now includes [The Glorious Glasgow--ridiculous, I know!] Haskell compiler.

Figure 1. GHC runtime

Next, I have the Happstack framework; Figure 2 shows the output of running vmc frameworks.

Figure 2. Happstack framework

Moving on, I’m going to deploy trivial Haskell/Happstack application. (Naturally, I could write some amazing Haskell code, but Hello World is the most didactical!). The source is

module Main where

import Happstack.Server (nullConf, simpleHTTP, toResponse, ok)

main :: IO ()
main = simpleHTTP nullConf $ ok "Hello, World! Haskell is COOL!"

Now that I have the application, let’s deploy it in CloudFoundry, using the usual vmc push command. The updated vmc recognises that the app is indeed a Haskell/Happstack one and selects the appropriate runtime/framework combination. (Viz Figure 3).

Figure 3. vmc push

Finally, we need to show you that the application is indeed running. So, on to the browser and go to http://haskellapp.vcap.me and see Figure 4.

Figure 4. The result

The code

Let’s take a look at what we had to do to get it running.

Hacking vcap

To make sure that the Cloud Controller understands the new runtime and framework, I modified the ‘app’ model by adding the following code:

Runtimes = %w[ruby18 ruby19 java node php erlangR14B02 python26 haskell]
Frameworks = %w[sinatra rails3 java_web spring grails node php otp_rebar lift wsgi django happstack unknown]
...
...
when "happstack/1.0"
    self.framework = 'happstack'
    self.runtime   = 'haskell'
end

Next up is the cloud_controller.yml file

  haskell:
    version: "[6-7].*"

Now we need make sure that the ‘dea’ is configured, so the dea.yml file

  haskell:
    executable: /usr/bin/runhaskell
    version: 6.12.1
    version_flag: '--version'
    environment:

So far the cloud controller knows about the runtime and framework and the dea is configured to deal with them as well. Now we need make sure that a Haskell/Happstack app can be staged in cloud foundry. We added the HappstackPlugin and the manifest for Happstack.

class HappstackPlugin < StagingPlugin
  def framework
    'happstack'
  end

  def stage_application
    Dir.chdir(destination_directory) do
      create_app_directories
      copy_source_files
      create_startup_script
    end
  end

  def start_command
    "runhaskell Main.hs"  #Equivalent of
  end

  ...
end

---
name: "happstack"
runtimes:
  - haskell:
      version: '6.12.1'
      description: 'The Glorious Glasgow Haskell Compilation System'
      executable: /var/vcap/runtimes/haskell-6.12.1/bin/runhaskell
      default: true
app_servers:
detection:
  - "*.hs": true
staged_services:

In this post, I’m ignoring the pains of installing Haskell/Happstack on Ubuntu 10.04–there’s a long blog post coming up about that soon! For the brave, take a look at setup/vcap_setup in the vcap repository.

Hacking vmc

Not a lot but still you would need the following to get vmc to understand the new runtime and framework: frameworks.rb

module VMC::Cli

  class Framework

    DEFAULT_FRAMEWORK = "http://b20nine.com/unknown"
    DEFAULT_MEM = '256M'

    FRAMEWORKS = {
      ...,
      'Happstack'         => ['happstack',
         { :mem => '128M', :description => 'Happstack Web Application'}]
    }

    class << self

      def known_frameworks
        FRAMEWORKS.keys
      end

      def lookup(name)
        return Framework.new(*FRAMEWORKS[name])
      end

      def detect(path)
        Dir.chdir(path) do
          ...
          ...
          # Happstack
          elsif !Dir.glob('*.hs').empty?
            return Framework.lookup('Happstack')

          end
        end
        nil
      end

    end

...
end

References & Downloads

We’ll send a pull request to the VMWare guys soon (after a bit of gold platting…), but in the meantime, you can fork https://github.com/anirvanc/vcap and https://github.com/anirvanc/vmc and get hacking!

I have got a perfectly valid set of arguments to justify “me not blogging”, but I would not want to bore you lot with that Instead, let’s talk about something interesting, like “Continuous Integration”! I am sure you all do it and so do I. We @Cake use Atlassian Bamboo as our CI tool, backed up by Ant + Ivy or Maven2 build infrastructure using JUnit. On top of that we have an excellent process of ‘Code Review’, which ensures that we deliver software of a very high standard. In other words, we are perfectly happy with our unit tests running overnight, showing a ‘green bar’ and developers addressing their code reviews first thing in the morning. I just love the ‘green bar’ though, but that’s besides the point. Now the question is “Is that enough of Continuous Integration?” or “Does that describe a fairly complete CI process?” I would have said “Yeah, thats totally enough”, till 28th January and I am sure many of you would still say that! But, things did change a bit for me since 28th January 2009, as I attended a webcast from DeCare Systems, titled ‚”Introducing CPM Toolkit ‚ Bringing Continuous Performance Management to JProbe”.

I have to agree, that I was a bit skeptical about the webcast. I was thinking in true Dilbert style, that “I am an engineer, I can’t listen to an hour of marketing talk”! But, boy oh boy, I was wrong! Not only did I like the webcast, I have also been taken by the “Continuous Performance Management” idea (honestly, it’s new to me). Jason Berry, the lead architect of CPM Toolkit did a wonderful demo. The tool is basically a combination of Continuous Integration and Performance Management. Which brings Continuous Performance Management Toolkit, a.k.a CPM Toolkit into play.

Here you can find a brief introduction of CPM Toolkit. Among others, the two points that got me hooked into it are, “Discovering performance problems when they are cheapest to solve” and “More time resolving and less time searching for the performance problem”. The CPM Toolkit allows you to pin point the performance problem at a unit test level, so you have a lot less code to find the problem in. This gives you the opportunity to quickly fix the problem and verify that your solution is not causing performance problems for any other parts of your system. I am pretty impressed with the GUI work the guys at DeCare has done. The graphical representation is simply impressive. To get a better idea of how CPM Toolkit works, I suggest reading Jason’s blog about CPM in practice.

One point I must mention here that I am still not 100% convinced that the use of yet another tool in the CI process will actually cut down development cost. Mainly because, I still need to see a real enough example of CPM Toolkit at play. So, I have decided that I am going to try out CPM Toolkit on my next project to get a better feel of it. I will definitely blog about the experience. In the mean time I would advice you guys to get a Free trial evaluation and give it a go!

I almost forgot that I still don’t have an answer to my initial question of “A perfect Continuous Integration process”. I guess, it’s very difficult to give a general answer which will fit all organizations. It’s more to do with the people involved in the process and the tools that are used. There will always be the next new cool tool, but it needs to be judged whether that tool fits into the process or not. I think it is fair enough to say that, introducing CPM ToolKit to your existing CI process can only make it better!

The need of the hour was to backup the live database and restore it on the test environment, which incidentally was also a Linux box. With pgAdminIII, it would have been really simple, take a backup of the database in a binary format and restore the test database from the binary file. So, off I went, digging into the PostgreSQL documentation looking for the commands which will do the same for me and wolla, I found pg_dump and pg_restore.

pg_dump is the utility for backing up a PostgreSQL database. Following is the command with it’s options that will create a binary backup of an entire database;
pg_dump -i -h host-name -p port-number -U username -F c -b -v -f "backup-filename.backup" dbname

pg_restore is the utility for restoring a PostgreSQL database from an archive created by pg_dump in one of the non-plain-text formats. Following is the command with it’s options that will restore the database from the binary backup file;
pg_restore -i -h host-name -p port-number -U username -d dbname -a -v "backup-filename.backup"
I noticed an interesting point while using pg_restore. pg_restore doesn’t work properly if there are referential integrity among the tables on the target database as tables are restored sequentially. So, the best restoring approach is to first create the tables in the database without enforcing their referential integrity, then run pg_restore to perform the magic. Once restoring is complete you can easily run another script to enforce all the referential integrities at one go.

On a completely different note, it is always considered as a best practice to test your application with a real data-set. This will give you the opportunity to write off those final few bugs which are only exposed when the application is used with a real set of data. What else could be better than the replica of the live database! More over, now you can backup and restore entire PostgreSQL database, even without the help of pgAdminIII.