Continuous Integration: A View into appendTo’s Process

Follow us on LinkedIn for our latest data and tips!

, , ,

Continuous Integration: A View into appendTo’s Process

Recently appendTo has been leveling up many of our internal processes, procedures, and standards. This has been a long time coming, but has been given dedicated attention recently. One of these efforts, the focus of this article, is appendTo’s most recent upgrade to our continuous integration and deployment (CI/CD) procedure and systems. This is one big piece of a larger standards and practices effort that is ongoing within our organization, and which benefits both our customers as well as our staff. If you would like to read more about our general developer workflow, standard practices, and tools you should read Aaron Bushnell’s recent article titled “The Tools We Use“!

Unique Requirements

There are some unique challenges to solve in a professional services organization when it comes to project server management. Each client, and each project for those clients, inevitably has unique software requirements. One client may have a back end in Rails which requires a specific application and testing stack while another may use Node.js with MongoDB. Furthermore, we could have one client which needs to use Node.js 0.10.28 because 0.10.29 has some change that breaks their application. These widely varying requirements result in appendTo not being able to simply deploy all client applications to a single staging server, we have to have unique environments.

Unfortunately, we don’t typically have the freedom to assign a server administrator to each project, and even if we could, that would not be the best use of our time, and thus our client’s investment. The solution that appendTo has implemented balances the needs of our clients with ease of use for our developer staff. The system is highly automated, but allows for extensive customization of the entire build process by the developers through simple JSON configuration within the project repository. We use TeamCity for version control triggers, build step management, access control, and reporting and logging. On the other side of the process we use Digital Ocean (and their HTTP API) to dynamically generate virtual machines (droplets) when needed and host and serve application files.

Bridging the Gap

The biggest problem we had with TeamCity was the ability to integrate, in the way we needed to, with the Digital Ocean API and individual droplets. We needed to create new droplets on the fly, allow each application to configure them during provisioning, deploy the application files, test them, and of course serve them up for our clients to see. TeamCity is written in Java, and while it may have been nice to integrate directly with that piece of this puzzle, we decided to create a solution which abstracts us one layer from that tool. That drove our engineers to create a bridge application which accepts commands (in this case from TeamCity) and executes code in our deployment system (Digital Ocean droplets) via a set of Node.js scripts and libraries. The benefit is that if either TeamCity or Digital Ocean need to be replaced in our stack, our unique build and deployment process can be salvaged and reused in the new stack. Of course, one of our greatest strengths at appendTo is our deep knowledge of JavaScript, which made Node a natural choice for this bridge.

So what does this all look like? The diagram below identifies the big pieces, on the left we have Github, where all of our application code resides. TeamCity, on the server in the center of the diagram, will identify code changes in a repository and initiate a build process. The build process will send various commands, in a particular order, to the Node application which in turn interacts with the Digital Ocean HTTP API and directly with the droplets over ssh. The droplets are provisioned when necessary (usually on first run) from a standard image that gives projects access to commonly needed software (like Node). Finally, the Node scripts report the results of each action to TeamCity in order to create a single source for build logs and history.

ci-cd process diagram

Wrap It Up and Put a Bow On It

This setup works very well for our needs, but we wanted to make things easier on our developers. To that end we made each step of the build process configurable within the project itself. Many of our projects use Grunt to run tests, concatenate and minify files, and generally build a project for distribution. In order allow for the simplest integration of that process, the specific actions required for each build step are specified in the “scripts” block of the project’s package.json file (a Node artifact).

Here is what that might look like:

    "name": "my-project",
    "scripts": {
        "provision": "apt-get install mongodb",
        "install": "bower install",
        "test": "grunt test",
        "stop": "forever stopall",
        "deploy": "grunt deploy",
        "start": "forever server/main.js"

Each of the lines in this “scripts” block maps to a step (or part of a step) in the larger build process. They will all be executed within the code directory, meaning developers don’t need to worry about where the code is located and can focus on what needs to be done to make it work. Additionally, since each line is executed (essentially) on the command line over ssh, the developer could specify a shell script should their process be overly complex for a one-line statement. Note that using Grunt is not at all required, but this process simplifies using tools like it with continuous integration and deployment.

Sustainable Process

As we discussed earlier, the goal with this bridge application written in Node is portability. Should we need to transition to a new tool on either side of that diagram above, we can reuse what we have written and our existing projects will require no alterations. This makes our developers happy, our clients happy, and results in better solutions delivered faster.

By implementing better standards and practices, and properly vetting those through our development staff, appendTo is able to accomplish two simultaneous goals: rapidly deliver better solutions to our clients and make the lives of our developers easier. The first may seem obvious, better standards (and adherence to them) directly translates into cleaner, more manageable code. The second may not be as obvious, but it is just as important to us. We value our developers’ time as well as their opinions. That’s why we vet our standards through them. Any employee is able to submit a change to our standards and practices. This change will be discussed among the entire organization and ultimately is approved by an S&P board which is composed of upper level engineers.