What is DevOps in an Embedded Device Company?
DevOps is a term that means different things to different companies or even different people and groups within the same company.
Are they the IT department? The build team? The automation gurus? Some even argue whether DevOps is a team of people or a concept applied across all organizations.
DevOps generally creates some confusion, but even more so in the embedded world.
In this article, I'll look at what DevOps is in an embedded device company and how we (yes, we have a DevOps team) work to make our company and its products better by enhancing how we work.
Formal Definitions
Let's start with various formal definitions of DevOps.
"DevOps is the combination of cultural philosophies, practices, and tools that increases an organization's ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market."
"DevOps is a software development methodology that combines software development with information technology operations. The goal of DevOps is to shorten the systems development life cycle while also delivering features, fixes, and updates frequently in close alignment with business objectives."
"DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology— especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective."
The Challenges of DevOps in an Embedded Device Company
Embedded products present a unique set of challenges to traditional DevOps. These challenges range from technical to workflow to culture.
Discontinuous Delivery
Starting with the backend of the DeOps workflow, the first challenge comprises 50% of the word itself -- Ops. We don't have standard "Ops," i.e., a production server farm, cloud, or otherwise, that hosts our code for thousands of users for which we employ continuous delivery.
We're a B2B technology provider and do not utilize a continuous delivery model with our customers. In fact, we employ a discontinuous delivery model over a matrix of software products and versions. Our software team delivers specific versions of code to specific customers, manufacturing vendors, and partners. Updates for features and bug fixes are a pull from customers rather than a push from us. Once in the field, our product software is NEVER updated again. We cannot benefit from blue-green deployment to flesh out features and bug fixes. Therefore, pre-delivery integration and testing are critical.
Continuous Integration and Automated Test
The test farm presents the next challenge.
Each commit or Pull Request creates the build package and then pushes it to a physical device for test to implement continuous integration. Software emulation with a scalable, elastic farm of containers is not an option. Therefore, custom hardware is necessary, and custom hardware requires physical space, physical connections, custom test harnesses and is difficult to scale.
Given the nature of our target platforms, we cannot take advantage of open-source web or test frameworks to develop automated testing. Therefore all tools and utilities are custom developed.
The lack of easy scaling requires an emphasis on automated and efficient test resource allocation and test time completion.
Which brings us to the build.
Build Toolchains
Cross-compile toolchains have been problematic from the dawn of embedded development because the target processor and OS (if one exists) are rarely the same as the host system. Embedded target builds are subject to the intricacies and nuances of the system on which it is built.
Long-time embedded developers have created a skillset around installing and maintaining these various toolchains purely out of necessity. But even with skilled maintainers of personal systems, every organization experiences the heterogeneous environment problems of "it won't build for me" and "it won't build for production."
Maintaining and configuration managing the embedded build toolchain is an organizational and technical challenge throughout the industry, and its no different for us.
And don't get me started on toolchain licensing...
Security and Safety Regulations
Many embedded products are subject to a set of security and/or safety regulations based on their industry.
Safety regulations tend to require adherence to a backend process of formal review and test, sometimes externally. Security regulations tend to require adherence to protocols such as encryption, location (people, systems, tools, and data), and access control.
Recommended by LinkedIn
Any standard or regulation that adds and enforces external or artificial workflow protocols works against the core DevOps concept of speed. We're in the biometrics and fintech space, which subjects us to security regulations.
Workflow and Culture
Although Agile and iterative development have made their way into the embedded world, the culture doesn't automatically embrace modern software development concepts such as DevOps.
Embedded developers are experts in registers, the stack, memory maps, and know GDB commands by heart, but many don't understand (trust?) the cloud, virtualization, and automation. Similarly, many web and application developers who work in the cloud all day do not understand what the heck is happening in the hardware (what hardware?).
Embedded developers are wilderness explorers, mountain bikers, self-sufficient, and tune their development machines as an Executive Chef cares for his knives. They are unimpressed by your fancy new processes and terminology.
In an embedded device company, DevOps requires marrying the two worlds -- finding the nuggets in modern software methodology that provide value in the embedded space. This marriage necessitates change, and change is rarely welcomed. The challenge requires a combination of technical skills in DevOps, people skills of influence and persuasion, and self-awareness to evaluate the usefulness of implemented changes continuously.
The Song Remains the Same -- Increase Speed, Quality, and Scale
DevOps aims to increase the speed and quality of deliveries and increase the organization's scale through efficiencies and automation.
That is our goal each day -- make our company and its products better.
We do so by applying standard DevOps techniques to a non-standard application. We also develop and employ some ambitious custom tools and workflows because of the specific challenges in our embedded space.
We have a philosophy, "make the right thing to do the easy thing to do." We do so by marrying human psychology with modern DevOps, but it takes experimentation and the willingness by both the organization and the individuals to fail. The key to making forward progress through failure is by learning from it and quickly adapting.
What does Continuous Delivery Mean?
Even though the external delivery process is discontinuous, we employ the modern DevOps concept of CD internally. In our case, the delivery endpoint is the test farm rather than a customer or external entity. We treat each CI build as if it will be deliverable.
As such, the organization has the opportunity to decide if any particular build can be deployed to a customer, vendor, or partner. This decision process, sometimes fully automated, sometimes not, determines which release makes it to QA and ultimately externally, but is only possible because each build could be deliverable.
Continuous Integration and Automated Test
To treat all CI builds as potentially deliverable, we must include a full "build => package => test" cycle. This requires efficient handling of cloud and physical resources to minimize time and increase speed. Therefore, we first decide what development activity receives CI. All commits? All branch activity? PR's only?
The CI pipeline itself is built upon a cloud foundation that most DevOps engineers would find familiar. We utilize a standard toolset for an elastic and orchestrated farm of containers behind the standard CI executor.
The test farm contains an array of custom physical devices into which we must push our code. Pushing and testing occur almost continuously as per the CI decision. To support this workflow, we've had to bridge the "Cloud to the Lab," and doing so in a secure environment. Build and packaging occur in the cloud, and then we push the code from the lab to the correct target systems.
Build Configuration Management, Tracking and Recreating Previous Builds
Embedded software deliveries typically employ a "hardware-like" versioning and serialization process. The software package(s) is often given a part number and included in the Bill of Materials (BOM) and the Product Lifecycle Management (PLM) tool. Because, like hardware, most embedded firmware is static, and knowing who has what version and what that version includes is critical to a healthy operations workflow.
Traditionally, organizations use (semi)manual Configuration Management (CM) processes to keep this straight, and it must start at the build. The build manager manually creates a golden-standard build on the special build machine and packages it correctly for the intended target. Then the associated CM metadata such as commit ID's, library versions, etc are added along with test results and documentation. This is, by definition, a process that disconnects the everyday work of develop, build, and test from the process of release and configuration management.
We've turned this on its head in our organization by ensuring that every build produced by our CI system is potentially deliverable. To do so, we have created a tool that automates the configuration management of the entire build and package workflow starting at the build and uses the production build environment (ie, containers) on the developers' local systems as well as in the CI system. Codenamed "Fuze," this tool is a build executive that wraps and automates build execution, dependency management, release management, and all required metadata attachment. A Fuze-built package is 100% traceable and reproducible on any system, not just the golden build machine, because all machines can build for production. The included constituent parts and dependencies are also 100% traceable, allowing complete forensic analysis of deployed versions.
Making the Right Thing the Easy Thing
If there was one single implementation principle that we try to employ in order to meet the goals of speed, quality, and scale, its "make the right thing the easy thing." Whether we're talking about people or systems, our goal is to focus on what is the right thing to do and attempt to make it the easy thing to do.
Most of the time, people choose the easy thing to do, regardless of whether it's right. That's not judgment; it's reality. So we endeavor to marry the two. When successful, the workforce is happy, the development speed improves, the product quality improves at all times, and the organization can scale accordingly.
We do all of this because those are the right things to do to help the organization go faster, with higher quality, and scale its capabilities according to its business.
Using "How We Work" to Enhance "What We Build"
The DevOps team continues to focus daily on how work is done. We look at the challenges on the plate right now or the challenges that we know are coming. Then we ask ourselves what we could put in place to make the how of that challenge efficient and effortless.
Increasing the speed, quality, and scale is an exercise in getting better at the how work is done. If a tool, whether open-source, COTS or custom-developed, helps how we do work, then we evaluate it. The more we focus on how we work, the better the products become.
Summary
DevOps in an embedded device company is, at its core, simply a term that means enhancing what we make by enhancing how we work. That's no different from DevOps in any company. Embedded systems provide some unique challenges, but DevOps can be applied with the proper mindset and skills and produce excellent results.
DevOps is like the grease in the system -- making the right thing to do the easy thing to do.
Infrastructure at Temporal
3yThanks for sharing John, this was a great read! Keep them coming!