This blog post can be considered a follow up of my previous post regarding unit-testing. In this post, I share my experiences and ideas for continuous delivery (CD) systems.
Wikipedia has a great sequence diagram of what the delivery pipeline looks like.
Jez Humble's Continuous Delivery blog has a great entry summarizing what CD is and its benefits through a series of cartoon images. I often refer to this entry to give people a quick summary of what I'm referring to when I mention CD.
So in this post, I describe why I am a proponent of CD and I share my past experiences involving CD and what it looks like in certain development environments.
I believe that the benefit of knowing the state of your software at any given time far outweighs the cost of setting up continuous delivery. Once you pass the hurdle of setting up such a system, you will be able to make software changes with confidence that all errors will be reported early before it becomes a nightmare to fix.
Essentially, you will need a version control system (VCS) and a continuous integration system (CIS) in place in order to facilitate CD. Ideally, you want to minimize human intervention in the early stages of the delivery pipeline. You want to make sure that all changes and promotions are recorded, which is where software change management (SCM) comes into play. All of these combined can result in a streamlined software development and release process.
I believe that VCS and CIS are required in order to make good use of CD. You need to be able to keep track of which versions are being used, which is where VCS comes in, and you need to package your software in a deployable manner with confidence, which is where CIS comes in.
I believe that a trait of any decent VCS is the ability to branch and merge codelines. You need to allow developers to continue their work as the software development progresses. A good development process allows developers to work independent from each other, which is often the case when the developers of scattered throughout the world -- open source software development has ample experience on making this process work.
Even with smaller organizations, you want to be able to protect your work from others, which is why you branch and merge. For CD, I'm accustomed to the idea of maintaining release and staging codelines so that developers understand which version of the software is considered ready for deployment.
Starting at Main.0 (the left-most node), the codeline is branched by a developer in Devel.0 so that independent work can begin in that branch. Meanwhile, Main.0 has been promoted to Staging.0, which too is a branch.
Staging.0 was evaluated and accepted for release and thus, the staging codeline is tagged (or trivially branched, depending on the capabilities of your VCS) to promote Release.0, which represents the version of the software that will or has been deployed. This version of the software is equivalent to Main.0 at this point -- note the colors.
The developer makes changes to the codeline and submits them to the development branch as Devel.1. The developer then wants to release this change so merges the new code with mainline to form Main.1.
Main.1 is then merged with staging to form Staging.1 for evaluation. In this scenario, the changes were not acceptable and so the changes won't be promoted for a release.
The developer then revises the initial changes as Devel.2 and then attempts to promote the changes for release by merging the revised changes with the mainline to form Main.2.
Main.2 is merged with staging to form Staging.2 for evaluation. This time, the changes are accepted and are promoted for release with a merge with the release line for form Release.1.
Meanwhile, the developer is able to continue development, as illustrated with a change in the development branch marked as Devel.3.
Another important prerequisite for CD is to have a decent CIS in place. You need a centralized build artifacts producer. Continuously integrating the staging version of the software should happen autonomously whenever a change is checked into the codeline. Basically, whenever someone merges their changes into the staging codeline, the CIS begins building the software.
CIS builds normally includes quick testing as part of the build process. If a test fails, then the build fails and thus, the changes are considered to be rejected because a broken build should never be promoted. When the build succeeds, the build produces some artifacts that can be further tested. If the software passes user testing, then it can be a candidate for a promotion to be released.
When a change is submitted to the staging codeline, CIS kicks in and attempts to build the software in the staging codeline. If successful, the build artifacts are available for user testing. Otherwise, the release process stops here and additional changes are necessary before the release process can resume.
If the user testing determines that the software is at an acceptable state, then the version in the staging codeline that was used for the user testing is then promoted to release. That process can simply be tagging the accepted version for release and then proceeding with the deployment.
If the user testing rejects the software at the state it's in, then the release process stops here and the process of checking in a revised change, building it in the staging area, and then testing is repeated.
In my experiences, there are three types of release triggers:
I now share my experience with these types of releases in a CD environment.
If a devastating defect is discovered in the software that is causing significant harm such as incurring cost, then an emergy hotfix release can be scheduled.
In these situations, the normal branching and release process is followed except the developer branches off the staging codeline instead of from the mainline or another development branch since the hotfix is meant to address an immediate defect discovered in released software.
This process is illustrated in the following code lifeline:
When the developers fixes the defect, the branch is then merged with the staging environment codeline and the noremal release process follows.
In my experience with sceduled releases, a cut-off point is determined in the staging codeline. That is, a change is designated as the last feature change accepted and so that particular build is known as the release candidate.
At this point, thorough testing is done on the release candidate.
If an issue is found, the development branches off of the staging codeline and the fix is merged directly back into the staging codeline. This process allows feature development to continue while preserving the designated feature cut-off of the release candidate.
Once the fix for the release candidate is merged and revised candidate is accepted, it is then released. The changes made to fix the release candidate is then synchronized with the development branches.
This process is illustrated in the following code lifeline:
I am a strong proponent of understanding the state of your software and I believe that continuous delivery is a great methodology to come to this understanding as it takes in good practices of development, testing, and requirements processes.
What I presented in this post may not work with your environment but I believe most of the required aspects of CD is in place.
Essentially, you need to maintain good branching practices to make sense of the evolution of your software and maintain your knowledge of the state of your software. While establishing CD may have some initial startup cost (getting VCS and CIS systems in place), the cost of not understanding the state of your software is much more devastating.