Posted by & filed under process, testing.

Start

What did we have?

We use Chef for infrastructure management at Safari Books Online. We have a Production Chef (ChefProd) server that performs standard tasks such as putting users on machines and setting up monitoring. Chef cookbooks also perform specific tasks such as setting up web, database, and application servers. In addition to the Production Chef environment, we also have Chef Development (ChefDev) environment used for testing Chef cookbooks.

We run a Gitlab server for version control. Gitlab is the open source project for self hosted git management. It is a git server and web UI written in Ruby on Rails and it works great.

Objectives

What did we want?

  • Goal #1: No knifing in anything to production manually.
    • This was for fat finger errors as well as a lack of version control backup.
  • Goal #2: Everything in Chef was in version control.
  • Goal #3: To have our entire Chef ecosystem within a testing framework.

Implementation

How we did it.

We built a Jenkins VM (provisioning users and monitoring through Chef) for our CI (Continuous Integration) testing server. Jenkins is a great tool for testing out software.

Each Chef cookbook is stored in version control with Gitlab. For each git cookbook repository, we made an identical Jenkins job. This Jenkins job would run all the tests on the cookbook on one of its nodes, searching for bugs. We added a git hook on merge to master for each cookbook. This means every time one of us git committed and merged to master, Jenkins (who was always listening for the Gitlab hook) would fire off the cookbook test.

All this happened in the Chef Development (ChefDev) environment and we wanted to accomplish Goal #1, which was put tested code into production without using knife. We needed a way to push changes to production, the simpler, the better. We arrived at having a trigger file, i.e. a file that would send the signal that we wanted to push to production. If and only if this trigger file was git committed and merged to master, Jenkins would deploy to production. We decided to use a CHANGELOG file (with the added bonus of frictionless deploy comments) but any file would suffice.

With this system, we could test out cookbooks on the ChefDev environment, and have Jenkins knife them into production at will. All code was in version control and tested. This fulfilled Goal #1, Goal #2, and Goal #3.

Vagrant

Vagrant makes development better. We use the above workflow and leverage Vagrant VMs for local testing before this entire process. With new cookbook development, it’s nice to be able to move quickly into writing tests and software. This is where Vagrant shines! We made special Vagrant VMs that mimic our production environment. DevOps can test new cookbooks locally before creating Jenkins jobs and Gitlab hooks. Existing cookbooks can be tested locally before the Jenkins tests, so you can develop and iterate quickly when starting out on a new feature.

Tags:

8 Responses to “Continuous Integration for Chef with Vagrant, Jenkins, and Gitlab”

  1. John

    This sentence is completely incorrect and a very bad statement to make. “Gitlab is the open source project run by Github which works in a similar manner as their site.”

    GitHub does not operate or manage anything related to GitLab. GitLab has it’s own team and none have ever been employed by GitHub. There are not associated in any such manner.

    Please correct your statement or ask GitHub for an official statement about their “involvement” with GitLab. https://help.github.com/contact

  2. Scott Pustay

    Hi Ben,

    We wanted to keep this system simple and sensible, we knew we would enhance and further automate it once it was running. This is exactly what happend, I’ll tell you how the current system works but we’re already working on enhancements to further automate the workflow.

    To keep things simple, we wanted to tackle individual cookbooks. We had each cookbook as its own project in gitlab. We then had an associated jenkins job for this cookbook. There is also a second general jenkins job (we called ours the integration-test) that ran tests.

    The goal we were trying to accomplish was everytime we ‘merged to master’, jenkins tests would kick off. On the gitlab side of things, we set commit hooks to post to the jenkins server. The jenkins server would listen for the commit hooks.

    The cookbook specific jenkins job would do the following when it heard a commit hook from gitlab.

    • Make a lockfile. (so two tests didn’t overlap)
    • Check which files were ‘merged to master’. (I’ll explain why later)
    • Fire off the integration-test job passing in the cookbook name as a variable.

    The jenkins integration-test job would be called with the variable of the cookbook name. This job would then:

    • Git pull this cookbook into it’s local filesystem.
    • ‘knife cookbook upload’ to the right test server. (based off of the cookbook variable name)
    • Run the tests and report the output.
    • Unlock the lock file we set in the first job.

    This told us if tests passed on a per cookbook basis. We also wanted the ability to push to production without knifing anything into production. We decided to use a trigger file that if and only if that file was merged to master, and jenkins tests passed, we would push to production. We choose to use the CHANGELOG file for the trigger file. So in the above jenkins integration-test list of actions, it would also ‘knife cookbook upload’ to the production server if tests passed and if and only if the CHANGELOG was the only file merged to master in gitlab.

    This system works great. We can put tested cookbooks into production without manually knifing anything in. The downside to this simplicity is setting up a jenkins job for each cookbook project is a little cumbersome. We are working on further automating this system as well as integrating other chef tools that will help with cookbook and dependency management, as well as better testing suites. I’ll save that for another blog post.

    Scott

Trackbacks/Pingbacks

  1.  Using test-kitchen with Berkshelf, LXC and chef-zero | Safari Flow Blog
  2.  Continuing on Continuous Integration | Safari Flow Blog
  3.  Building Vagrant VMs with the VMWare Fusion provider | Safari Flow Blog
  4.  Ramblings 2 @ Java | Bemused