How To Prepare For When A Key Employee Gets "Hit by a Bus"

I'm not sure where it came from or how it started, but that's the euphemism we use. As in, "Can we still deploy if Joe is hit by a bus?" It sounds either flippant or morbid, but really it's just short hand for anything that could make someone unavailable with little or no notice. That includes an accident, but also something like handling a family emergency, quitting, getting fired or even retirement. (Congrats, Joe!)

Are you prepared for your key employees being taken out of the loop? Is your company prepared for YOU being hit by a bus?

Dealing with something like that is never easy, but with planning and on-going preparation, you can make it much easier for the company/project to survive.

Passwords

The immediate thing that comes to mind is passwords and access to accounts. This is especially true if Joe is an administrator of any systems. The key here is to use a password manager that is secure, but also has distributed access. Password managers are also the number one way to start improving password security, so this is a no brainer!

Mandate that everyone use a password manager, and make sure that it's either an online service or something that is backed up regularly to somewhere that can be accessed. Make sure that the service or process is encrypted and secure; you don't want this to become your next security vulnerability. An excel spreadsheet of accounts copied to a network drive is not a good plan!

I personally don't like cloud services, and I like the Keepass application, so I use a self-hosted web solution that has a Keepass interface. The actual database is backed up to a zero knowledge encrypted system, which more than one person has access to.

Of course you also need to consider what happens if your password server gets hit by a bus! You need some sort of (secure!) redundancy here to. I found this out the hard way when I couldn't access my server password to fix an issue with my password server! I now keep a copy of a few critical passwords in my personal Keepass repository (which is also backed up, and which a few key people have access to).

Multi-Factor authentication

Multi-factor authentication (MFA) is where a login system also sends you a text or email to verify your identity. MFA can increase security, but it needs to be used with care. Ideally the second factor should be something the company has access to, so it shouldn't be a personal email or personal phone. Even a work cell phone could be a problem; if the phone itself is hit by a bus it could take you days to get the number transferred to a new SIM card.

MFA isn't such a big deal on most of Joe's logins, but imagine if he's the technical contact for your domain and you can't login to renew your domain or update DNS entries without receiving a text on Joe's personal phone!

Some systems allow a secondary "recovery email" for resetting passwords. Setting that to a company address that is secure (i.e. only a few people can access) can be a good plan.

Email

The goal here is for all of Joe's work correspondence to go through joe@yourcompany.com instead of his personal email account. This is true for logins, but also for emails to clients, vendors, or whomever. You should be able to access Joe's email and redirect it if needed, so you don't miss any critical communications.

If you have access to his email, that may also help you reset the passwords on any accounts where you can't recover the password.

Emergency Contact Info

This one is pretty low tech, but if you really need an employee's laptop or work phone, it's pretty hard to get it without someone to call! Get this info from your employees and have them review it once a year to keep it current.

Source code

I hope this goes without saying (but I've worked with enough variety of clients to know that it doesn't), all source code should be in a source repository and kept up-to-date. Make sure that more than one person has full access to this repository, and that nothing gets deployed without being in the repository. (That excludes secrets like database passwords and authentication keys. If you have secrets in config files they should be in your encrypted password repository or other solution, not in source control.)

There are plenty of cheap or free online solutions for source control. There are self-hosted solutions as well. Github is pretty commonly used and well known among developers.

Documentation

Having code documentation is important, but even more essential is high-level and process documentation. How does our software get deployed? Where are our servers? What are their purposes? How does data flow through the system? How do we add a new client to our SAS product? Good documentation can help answer all of these. Having the docs in a wiki or other online solution can make them easier to update. You should have a task reminder to review this periodically and refresh documentation.

Hardware

Does Joe have any unique or hard to replace hardware or physical systems? Some examples of this might be having his IP address whitelisted to administer a system. Or he has a special USB dongle for accessing a vendor product. Or he's the only one with the prototype of the new product we are building.

These can be hard to eliminate in some cases, but you should at least take inventory and document them so you aware of the failure points.

Code Readability / Maintainability / Quality

This is a special concern if Joe is a developer, and one of the hardest to address. If you have multiple developers on your team, you ideally want to avoid silos and sandboxes, where only Joe works in this area, so only he understands it.

The overall goal is to make sure your code is high quality and maintainable, so that you could hire another developer to take over after Joe. This isn't just a quality issue though, you also want code that uses languages, libraries and techniques that are commonly used and understood.

I'm highly cognizant of this. As a contract developer I want to be invaluable but not irreplaceable! I try to avoid languages and platforms that don't have broad commercial use. This can be regional too; in some areas it's easy to find a Ruby programmer, but it's pretty rare where I'm at.

This doesn't mean that I code for the least common denominator, and that all of my code can be taken over immediately by a fresh out of boot camp developer! Rather, that a seasoned dev should be able to recognize what I'm doing and understand the code at a high level without much effort. I evaluate each library, language and framework from this view point; does it bring enough value to justify its cognitive load and learning curve?

This is a big concern for legacy systems. Over time it gets harder to find developers who want to work on, or who have experience with, really old technology.

Conclusion

I'd like to say that surviving being hit by a bus (you or someone else) is easy, but it won't be. Preparing for that eventuality needs to be a constant on-going task and part of your day-to-day processes. Make sure to do a test run of your recovery plans periodically! See if you can access your source repository. Log in as an admin to some of your critical systems. What happens if your password system is unavailable? Good luck, and please look both ways before crossing the street!

This is a technical/business article catered to developers, hiring/project managers, and other technical staff looking to improve their skills. Sign up to receive The Informatizer in your email inbox.

If you're looking for a job in the tech industry, visit our job board to see if you qualify for some of our positions. If you're looking to hire technical talent for your company, please contact us.

Stout Systems is the software consulting and staffing company Fueled by the Most Powerful Technology Available: Human Intelligence®. We were founded in 1993 and are based in Ann Arbor, Michigan. We have clients across the U.S. in domains including engineering, scientific, manufacturing, education, marketing, entertainment, small business and robotics. We provide expert level software, Web and embedded systems development consulting and staffing services along with direct-hire technical recruiting and placements.