I love automation. There are only so many hours in a day and automation gives you a lot back. This is also why I love continuous integration and deployment. CI is the practise where software developers upload (‘commit’) their code changes to a central repository at least once a day. Each commit is then verified by various automated tests to make sure that it’s working properly. Continuous delivery is the natural extension of this where deploying the code into the production environment is also automated.
This is why when I decided to drop Wordpress I wanted to replace it with something that would allow me to automate the whole process. One command to commit, build and deploy? Yes please! In the end I decided to use a static site generator (Jekyll specifically) as they fit the CI/CD process very well. You add the layouts, content, etc and then build the static site from them. All that’s left to do is upload the new site.
The overall process I set up is dead-simple. Every commit triggers a post-commit command that sends a notification to Jenkins. Jenkins fetches the latest commit and then builds it. If it fails I get an RSS notification and if it succeeds it copies the new site over via scp and notifies me via RSS. It takes about 10 seconds from start to finish.
Setting up Jenkins
Jenkins needed 3 plugins for this project:
- Git Plugin
- Bitbucket Plugin
- Publish Over SSH
It was originally a toss-up between Bitbucket and GitHub as both support post-commit webhooks that can notify Jenkins. However in the end Bitbucket won simply because I use it more often (free accounts get private repos).
The plugin configuration was straight-forward. The Publish Over SSH plugin was already installed so I already had an SSH key I could copy over to the web server. All I needed to do was add a new “SSH Servers” item which can be done via Manage Jenkins -> Configure System. Finally, as my repo is private I needed to configure a set of credentials for Bitbucket Plugin to use. This was done in the ‘Credentials’ section (accessible from the Jenkins dashboard).
Next, I created a new free-style project. The most important parts are as follows:
Source Code Management Section
Here I chose a Git repo and put in the URL. It’s the same one you use to clone it however you should remove the username if it’s there. Next, I chose my Bitbucket credentials as it’s a private repo. I also added the “Clean Before Checkout” behaviour from “Additional Behaviours”. This deletes any files created from a build and restores the workspace to the previous commit before doing a checkout.
Here is where all the important stuff happens. I use an “Execute Shell” step that contains all the build commands. What’s important to note is that Jenkins stores the commands in a temporary file before executing them. Thus, my script looks like this:
#!/usr/bin/env bash [[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" rvm use 1.9.3 rvm gemset use linuxydave sed -i 's/localhost\:4000/davidsj.co.uk/' _config.yml sed -i 's/^#disqus_sitename: davidsjsite/disqus_sitename: davidsjsite/' _config.yml jekyll build
Let me give a quick breakdown for those who don’t know what the hell any of that means:
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" rvm use 1.9.3 rvm gemset use linuxydave
The above code sets up the RVM environment. As Jenkins runs the commands in a script file it was very important to add the first line as that it loads up the RVM environment variables. If you don’t the build will fail with the error message “RVM is not a function”.
The next two lines are rather self-explanatory if you have used Linux before but I want to go into a bit more detail:
sed -i 's/localhost\:4000/davidsj.co.uk/' _config.yml sed -i 's/^#disqus_sitename: davidsjsite/disqus_sitename: davidsjsite/' _config.yml
sed is a command line tool for manipulating text via regular expressions and
I’m using it to update two of the settings for the live site. By doing this I
never need to worry about forgetting to change them and, more importantly, they are never
stored in the repository. It may not be important to keep the two environments
separate in my case but is a good habit to get into.
Simply put, you shouldn’t have production settings in a repository, especially
if they are security-related. It’s a no-brainer but you wouldn’t believe the
number of developers who do it!
The final command is easy to understand - it builds the site. Not much more to say about that.
Post-build Actions Section
Once the site is generated it needs to go somewhere. The Publish Over SSH plugin adds a new post-build action, “Send build artifacts over SSH”, which is what I use. While you can only have one “Send build artifacts over SSH” action you can have an unlimited number of transfer sets inside it. For this project I chose the SSH server I added in the beginning and added two transfer sets - one to delete the previous site and another to copy the new files over.
Transfer Set 1: In this one I only use the “exec command” option which allows you to run a remote command via SSH (in this case, “rm -fr path/to/site/*”).
Transfer Set 2: Jekyll stores the site in “_site” so for this set I use an Ant pattern to specify the directory and files to upload (“_site/**”) plus a “Remove Prefix” of “_site” so that only the files are copied across. Otherwise, “_site” would be copied into the web root, ie, path/to/web/root/_site. Lastly, I set the “Remote Directory” to the web root (relative to the user home directory).
Set up the Bitbucket Webhook
With that out of the way we can set up the webhook. On the bottom-left of the repo overview is the settings button. From the settings page I added a new Jenkins hook and entered the endpoint (the publically-accessible URL of my Jenkins server) and the project name (which is the same as the Jenkins one).
As my Jenkins server is in the cloud I’ve firewalled it off very heavily. The final step was to allow the outgoing Bitbucket IPs through my firewall. Currently, they’re ‘126.96.36.199’ and ‘188.8.131.52’.
Now that there’s a continuous integration/deployment system in place I’m updating the site much more frequently (which is kind of the point…). Currently the updates are focused on porting the BT3-Flat Pelican template and adding features however I can see more potential in the system. For example, why not use Jenkins to implement some social media functions like updating Twitter or Facebook? I think that automation’s biggest benefit is the flexibility it provides if you keep an open mind.
Delete Queues In RabbitMQ