The Methods of Producing and Deploying Artifacts for the Three Main Software Manufacturing Methodologies
Understanding how lifecycles differ is to understand their uses and how to deploy the artifacts that are a result of these lifecycles. As there are no industry best practices to fall upon to fully understand what the different lifecycles can offer an organization is to understand the artifacts produced and how they are treated when deployed into an enterprise’s production environment. For lack of an industry standard term, the term “Gold Build” will be used in referring to the production and staging of these artifacts up to deployment to the production environment.
In order to perform what is called Gold Build builds by many organizations we first need to determine the type of lifecycles that software artifacts are manufactured in as each lifecycle will carry both a workload and a risk factor that needs to be considered, such as business and technical planning, development efforts, technology viability, and risks of release that ultimately come down to how the artifacts are built, packaged and released.
What are the common lifecycles that the general software industry uses?
Before moving forward, there is a compelling need to understand what the three types of lifecycle methodologies that currently use. These are the SDLC (Waterfall), Agile, and DevOps methodologies. To define these, it is best to step away from Software Manufacturing and look at manufacturing in general. The automobile industry is perhaps the best and easiest way to see the differences of the 3 methodologies so that they make the most sense for everyone. It should be understood that this is not an in-depth analysis in which methodology is better, just they are different and have a specific place in Software Manufacturing in general.
Understanding the Lifecycles and Methodologies
SDLC
The SDLC is concerned with the overall delivery of the project. Even if it consists of many multiple parts, the project is only valid to the enterprise if the whole project is delivered. This concept is most associated with creating an assembly line that produces cars. In this the assembly line needs to produce cars, otherwise it is of no value to the enterprise. While engineers may argue about the “best” way to make cars, the shareholders are only concerned about making cars, and in order to make cars the assembly line must be completed. The assembly line or SDLC may be composed of several tasks need to take place near simultaneously to create the deliverable which here is cars, such as the mechanics of the line, automation at certain stations, part feeds, and more. It is understood that the only changes that must take place are those changes that make the line work, changes to make line work better or even better cars are belayed to a later time. Once the assembly line is running, and the CEO drives the first car off the line, pictures are taken, and changes that make it more efficient take place.
The SDLC is principally concerned with an enterprise release, such as putting in a new architecture where it is understood that for the company to be profitable venture. The danger with an SDLC release is it can be taken over by well-meaning stakeholders and never get finished because of “feature creep.” Feature creep is trying to fix real or perceived problems before they are problems, and simply must be avoided until the release that the SDLC was commissioned to make. At that time teams can invest their efforts in to fixes and features to both the product and processes. This is normally done with an agile methodology.
What needs to be understood is the risk factor. The SDLC carries the highest risk as often it is not until the end that will be known if the enterprise’s efforts bore fruit or not. Prior to agile manifesto the enterprise needed to wait until completion SDLC to understand all the risks, or risking pushing back the release date further and further until the market made the effort meaningless. The SDLC’s timeframe typically can range from several months to even years to complete.
The future of the SDLC has been given a new lease of usefulness, today it is easy to see the SCLC can be comprised of several agile projects working together that are treated as a single long-term process that augment the risk. This new way of looking at the SDLC from a huge monolithic project to an aggregate of smaller projects lead by specialized teams has had the result of preventing the stopping the SDLC and the overall project, and concentrating on the single agile project where stakeholders can have their concerns addressed.
Agile
If SDLC is associated with manufacturing cars from scratch, we can look at Agile as fixing broken cars and increasing the performance of ordinary cars. When a car breaks down you do not build a new one, instead you diagnose the cause and replace the broken parts. Auto repair shops and performance centers take a car that came off the assembly line and either fix it, so it keeps running or modifies it, so it runs better.
A shop is not concerned with what has taken place on the assembly line, just with what is at hand. In so doing a shop does not need months to get ready, just a short phase of research or diagnosis, part procurement, part replacement, and testing. Agile project can take a week or more as it may require a research spike, several lines of code to change to fix a bug or implement a feature and testing or if may take a month or more to retrofit a new database into the main programs with the program is still running with the old database. In agile projects time is consumed by both unknowns and complexity.
The risks of agile are not as severe to projects and can be labeled as moderate. Often a research spike will prevent a team from expending their energy of solutions that simply will not work. The timeframe where agile should be used is several weeks to a couple days can be streamlined by automating several common tasks that are typically associated with DevOps such as CI/CD.
DevOps
If the SDLC makes a car from scratch and Agile can perfect that car that rolled off the assembly line, you may ask what DevOps is good for. Keeping with the automotive motif DevOps is like a business that changes your oil, tops off your fluids, installs or rotates your old tires.
Anything that can be done in minutes or few hours and has a zero-risk profile is an excellent candidate for DevOps. One thing important with DevOps is once it is released as a service there should be no research, and no waiting for expertise. Performing builds, running first line tests, changing the color of your homepage, archiving logs, running preapproved scripts that also have minimal risk are all candidates for DevOps.
At this point the explanation of the three methodologies and analogy with the automotive industry is over. What should be able to be seen is that the methodologies run from the most risk prone to the most risk free, from complex to the simplest, from the longest to the shortest to complete.
This does not mean one methodology is better than another, or that you must choose just one methodology. They can be separate for differing products or circumstances or used in conjunction with other methodologies to product a solution that offers the enterprise the best chance of success, with the less risk, in the quickest timeframe and at the least cost. DevOps can be used with Agile and Agile used with the SDLC to create a working process design that increases stakeholders’ confidence.
Each Methodology has it purpose depending on the need.
In the end choosing a methodology is like choosing what to do with your car. Replace it, replace the engine or change the oil. All may do the job, but cost and overhead need to be considered.
Deploying artifacts with the several methodologies
It is probably best to define what a “Gold Build” is. A “Gold Build” is a term found principally in the SDLC and carried over to the Agile software manufacturing methodologies that denotes the artifacts that were built, tested and segregated for deployment into the production environment,
In both the SDLC and Agile methodology because of the longer time frame to deliver releases are often built up. Features are added by one team, while fixes by another team, causing testing to take place and perhaps the integration of features and fixes requires a fix itself. Code is transferred from a chaotic development branch to a more refined and stable release branch. Each delivery to the release branch requires a new build to integrate the changes into that release. This has no correlation in DevOps as every individual build is treated as a (potential) Gold Build to be deployed into production as part of the pipeline.
What makes deployment different between SDLC, Agile and DevOps methodologies are that the release for SDLC and Agile are complex and are planned to happen on specific dates, that usually involve other teams and regulatory constraints. In DevOps, every build is potentially “releasable” the artifacts built are always staged for deployment and become nothing more than a transition between Constant Integration and Continuous Deployment (CI/CD,) as it is done via a single pipeline.
Another distinguishing feature of the different types of “releases” is the mechanism that actually does the release of the artifacts into production. Typically, in DevOps this release mechanism is done with a quality-gate that either automatically or manually approves the artifacts for release, while the SDLC, and Agile to a great extent, utilizes the CCB (Change Control Board) to release the artifacts.
How a release is made is a combination of the actual risk and the enterprise’s risk threshold and correlates with the SDLC and Agile methodologies a release is much more complex and in general the inherent risk is greater; DevOps is much simpler, and the risk is perceived as negligible.
The next step is to define what is meant by a Gold Build and to examine the mechanics and reasoning behind preparing for a gold Build in each methodology. As each methodology brings differences, similarities, inclusions with other methodologies, and dependencies.
Simply put a Gold Build is the preparation and staging of the artifacts created for inclusion into the production environment. How this takes place is dependent on the methodology the artifacts are created in.
SDLC Gold Build – As stated the SDLC methodology is primarily for an enterprise release. What is often overlooked is that the “release” may see huge, but often it is not monolithic and is recent days it is composed of several project or products that may have themselves used their own SDLC or Agile.
A SDLC release on the surface may be referenced as 3.2.25 but is composed of several products with their own releases aligned in a matrix. Because of its complexity with multiple products, it is called a Dependent Release, meaning that there are inherent if not explicit dependencies between the components. Below you can see a working example using a simple MVC project with a database.
What is important to understand while the overall Enterprise product (in blue) utilizes the SDLC methodology, in that construction and testing are dependent, components such the Model, View and Controller all use the Agile Methodology to produce the releases at different stages for the product. While the MVC is internal the external database is vendor adds dependency to Model and complexity to the overall product. Resulting in that all components need to be released in an integrated fashion.
We all accept the risk in stopping a SDLC project to add features or fix minor bugs but having the components in the agile methodology allows for acceptable and controlled risk and enhanced stakeholder feedback during the whole process.
In preparing for the release into production a separate area called “Gold’ is often used. It contains the Enterprise Release number followed by the components and release number and their artifacts. While it is the same top-level structure as an Agile release what differs is that multiple components are below and included it the SDLC release. It may seem redundant to have the same version in multiple areas, it is extremely useful, otherwise, the correct version is not inherent in the file structure of the component release, and the reliance is on a separate document.
The filesystem artifacts are stage in are not very different than with a filesystem used in an Agile deployment. Basically, the only difference is in segregation the deployment for the overall product separately from the several components. The major difference is that the release area is distinct from the area all artifacts are stored in. In the filesystem of a SDLC release there are no directories representing development branches with dozens of releases, nor are there release branches with separate builds that were never released. Hosting several components/products, with several branches and dozens of builds on each branch increases the complexity proportional to the error rate in choosing the wrong branch and version. SDLC staging areas hold just what is needed to release the project and it is the model for the DML (Definitive Media Library) of ITIL v3.
To note there are many Agile techniques in readying a SDLC release for deployment, such as restricted artifact locations and locked branches, but what is never seen in this methodology is a DevOps release technique such as DevOps Continuous Delivery. Again, this is because a DevOps CI Build used with a Continuous Deployment only releases into a live system. However, understand that DevOps techniques are valid, such as building with Continuous Integration in order to stabilize and quicken the desired line of code when tooling an Agile deployment.
Agile Gold Build –
Primarily a product or project release Agile release will look like the release of SDLC, although restricted in size.
But they are unlike the releases of SDLC that are stable and quiescent, Agile releases are dynamic and always in a state of flux, causing the Agile artifacts to have a life of one or two days.
In keeping with the automotive motif, the artifacts for deployment are arranged precisely in order, with just the right on the overhead parts line. Whereas Agile’s artifact deployment is half Part Store and half Junkyard. Teams are constantly developing on Dev branches and merging into Release branches for final release, and as a result, deployments may be going to several environments at one time. However, in test environments it is not uncommon to find 70% of the artifacts coming from production and 30% coming from Development and Release branches.
Risk is a constant factor with Agile deployments. The results are often not as serious as a failed SDLC deployment, which can bring the whole IT of an enterprise down, an Agile deployment can bring down one or several applications depending on the integration between the applications. The perception that Agile’s length of cycles makes development quicker, stakeholders demand a faster turnaround. This speed also increases the risk of the deployment, as it is easy to deploy from the wrong branch or wrong build number in Agile when all are available to deploy.
The complexity in the artifact storage area becomes relevant with the two diagrams. In the above diagram holds other directory that hold the type of release, while the diagram below shows the branches themselves for that type of artifact. Gold artifacts are production quality, while Release artifacts are stable and under the correct circumstance may be release as they may be lacking in testing and approval, and the artifacts drawn from development are simply too unstable to release into the production stream.
Inside the directories for the branches are directories correlating to the build itself. Typically called build numbers, they can be date/time stamps.
Difference Between the SDLC and Agile Methodology?
As noted above SDLC can and often does use Agile methodology although in a component level. This is primarily because a system that the SDLC was designed to implement is comprised of components. The difference lies in the complexity and scope. A MVC application is comprised of three components but may not be big enough for the rigors that an SDLC puts on a release. Likewise, a database change is only one product, but it can affect hundreds of applications leaving its release as a standalone dangerous. Whereas an enterprise release of multiple independent applications has scope, it does not have the complexity of installing a new database which in turn has a narrow scope.
In choosing to release under a SDLC or Agile methodology the factors of scope and complexity indicate the risk of the endeavor. If the risk is high, it would be best to use the SDLC approach of several independent products becoming integrate to become an enterprise product. Whereas if the risk factor is acceptable to the enterprise risk envelope the Agile methodology is appropriate.
In the below diagram the DevOps techniques of Continuous Integration and Continuous Testing are used to produce a candidate for release. At the end of testing phase, a manual decision is made based on the results; stop further processing and open a defect or accept Testing/Developments request for staging the artifacts for deployment.
There are several methods used to create a Gold build for either. During the build phase one technique is to label the build with a unique identifying label so the code with that label can be reused at a latter time. Another technique is to rebuild the release branch in a pristine environment, set to the correct runtime environment and that has no test hooks or anything that can interfere with runtime integrity. Another method is to simply to promote the artifacts used in testing to the staging area. What determines how a Gold Build is produced can be different for each application. There is also an applications legacy methodology which can be either so ingrained or so close to the practices derived in the methodology that it is continued as the chosen methodology to create a Gold Build, and often becomes the best practice.
Governance
In either case the SDLC or Agile, what separates both form DevOps is the Change Control Board (CCB). The CCB is a governance body within the organization that decides if a release meets the organizations profile for a successful release. The need is verified, plans are validated, and all documents and artifacts are accounted for. Clearly not all releases are successful, but it is the CCB’s to pause between the build, testing and release and look for obvious anomalies.
DevOps Gold Build -
DevOps does not have a Gold Build per se, in the DevOps mindset every build that passes the test battery is a canidate for deployment. That means DevOps is restriced in what it can deploy because what can be deployed must be preapproved.
Just as with the SDLC and Agile what is chosen must be weighed against the scope and complexity in order to fall within the acceptable risk to the enterpise. DevOps release carry no or minimal risk, are quick and always independent of other releases.
As seen in the diagram below, the build the artifacts go automatically to an automated testing reigme and to the Continous Deployment automation. The only event that stops the transfer are automatic and manual gates.
How a single pipeline manages both the testing and deployment of these artifacts is that the artifacts become a part of the build or Continous Integration and travel in the pipeline through the several components as an archive of the build job.
Typically, the build stores the artifacts as part of the build jobs internal archives and can hand references to these artifacts to the follow-on pipeline components, such as testing and deployment. As stated, there are no Gold Builds from which operations deploys from, but it is a general good practice to copy these artifacts to a storage location for redeployment, governance auditing and testing later versions against.
General Best Practices for Gold Builds
Best practices that are not determined the industry, must be constructed on solid footing. Their use, the environment in which they are too be used, and any regulatory factors may mandate the collection and care of the artifacts used.
While each methodology has its own needs in preparing the artifacts for production, what must always be addressed is the enterprise risk profile on top of what the methodologies would describe as basic care and preparation.
SDLC and Agile
1. The source code for all releases must come from a release branch create for that release.
2. The build must be built for the production environment.
3. The build must not have any test hooks that can impair its run time performance.
4. If the Build Verification Test is successful
a. The branch is labeled for each build.
b. the branch is locked so new work cannot take place outside of controls.
5. The artifact storage location should be labeled with the same nomenclature as the source is and can be used all conditions are met.
6. If condition 2 and 3 are not met, a rebuild is in order insuring that the build environment’s configuration is for Prod and internally the artifacts do not contain test hooks that can interfere with its run time performance.
7. The artifacts must be stored in a write restricted area that is owned by the designated authority and is read accessible to operations.
a. These artifacts can be stored in version control.
8. Merging of the Release branch into the “production” branch so the code is always up to date and prevents regression.
DevOps
1. All release must come from a generic release branch.
2. All manual gates that override automated gates that release artifacts into production must be recorded.
3. Source code used should be labeled.
4. All artifacts must be archived either in a protected filesystem or in version control. .
5. ** As DevOps development is highly specialized there cannot be no universal rule for merging back to source. Often development consists of configuration files in template form, modified per specific task and cannot be merged back to the source as it will destroy the template.
Closing Thoughts
In researching what to do with the artifacts created, ITIL 3, CMMi V2, and COBIT 2019 were consulted, they could offer nothing except to be considerate of the risk when injecting change into the production environment. Also, over thirty-one different software models were consulted, and they all had differing artifacts and needs. In the end there are no best practices, but the how and why software is made can point the way.
Methodologies are neither the start nor finish of software manufacturing, they are merely a means. Each methodology has strengths and weaknesses that can be exploited or avoided. If the SDLC is used, an organization’s requirement will have a 99% chance of being met, while at the same time changes the market may would make those requirements meaningless. Agile changes the SDLC to keep the business stakeholders in the loop, but they use many of the same manufacturing techniques. The industry’s hope for DevOps was that it would be the silver bullet management has been looking for since the first program was sold., but it has given teams not the silver bullet, but a machine gun to do more in the same time with ideas of Continuous Integration.
Select the methodology to use for the application and then worry about how the artifacts are handled.
What does shine through was that these artifacts could point to the cause of failure and would be become the primary witness for the team whose task is to fix the issue. Documenting not only the artifacts but also what creates them is of enormous benefit in this task.
Disclaimer: This does not reflect the views or methodologies of my current or previous employers. It is the amalgamation of acquired knowledge from nearly 35 years in the Software Manufacturing discipline.
IT Engineer Specialist
3yGreat Ted!!!😎👊🏾