As published in the September/October 2020 edition of InfoSecurity Professional Magazine
By Anita J. Bateman, CISSP
We are all plagued by technical debt in the form of legacy systems that can no longer be patched but must be kept up and running. Critical business processes, legacy data retention, lack of system knowledge or “pet” projects might keep us from retiring these difficult-to-maintain systems. From the very first operating system updates on the original IBM 360 to the latest Windows 10 updates today, we still struggle with this common challenge to fully patch and maintain our technical systems.
Might there be a different way to approach this perennial issue? Might we invoke some of the philosophies, principals and methodologies of organizational experts when it comes to ridding IT systems of so-called junk?
Patching alone will not solve this
How did we end up with so many unpatchable systems in the first place? Mergers and acquisitions have brought us systems that may not conform to our standards of maintenance and lifecycle management. “Shadow IT” and our own lack of discipline may be additional sources. Ongoing challenges to “do less with more” may have forced us to make prioritization decisions to defer regular maintenance activities on lower-priority systems.
We know that we can improve the situation with well-defined processes, dedicated teams, smarter tools, more budget and better discipline—all the usual best practices. But it is rarely this simple.
Some industries, like manufacturing and healthcare, have a harder time with regular patching schedules because downtime is nearly impossible to schedule.
Dr. Marianne Winslett, computer scientist, is familiar with this. “There’s a reason they [factory floor] never patch and never upgrade. It’s because any time you do that, you’re at significant risk of downtime. … A factory that’s running 24/7 really can’t afford the downtime that comes with computer system failure. So, there’s an ‘if it ain’t broke, don’t fix it’ attitude.”
NIST has a project underway for “patching the enterprise” in its Critical Cybersecurity Hygiene area.
The final project description, published in March 2020, says the “objective of this project is to demonstrate a proposed approach for improving patching practices for general IT systems.” As this study is only addressing general IT systems, it will not help in tackling business applications or other specific topics, like Internet of Things (IoT) devices, but it may provide a useful reference guide.
Other traditional approaches to solve this problem have focused on rationalizing your business application portfolio and modernizing legacy systems. In a 2018 titled “7 Options to Modernize Legacy Systems,” Susan Moore explains how to choose between options to encapsulate, rehost, replatform, refactor, rearchitect, rebuild or replace. However, we may not know enough about a legacy system to take on a modernization project. “Not only are the people who built these systems gone, the users who asked for them to be built in the first place are also gone,” writes William M. Ulrich in Legacy Systems: Transformation Strategies . (See sidebar on p. 21 for additional patching and legacy modernization resources.)
New Approaches
What if we approached this problem as a true cyber hygiene problem, similar to how we approach cleaning our homes, our garages and our communities? Our homes and garages accumulate “junk” just as our data centers accumulate technical debt. Similarly, a system upgrade or transformation project could be viewed as a community project with many interested individuals—including business owners, financial stakeholders, end users, IT professionals and executive leaders. What insights can be gained by approaching our unpatchable systems in this manner?
Three resources provide such an opportunity: the KonMari MethodTM ; garage organization tips from Family Handyman; and tips for a successful community cleanup.
Marie Kondo to the Rescue
Let’s consider the popular KonMari Method invented by Marie Kondo, which “encourages tidying by category—not by location—beginning with clothes, then moving on to books, papers, miscellaneous items, and finally sentimental items.”
There are six basic rules:
- Rule 1: Commit yourself to tidying up
- Rule 2: Imagine your ideal lifestyle
- Rule 3: Finish discarding first
- Rule 4: Tidy by category, not by location
- Rule 5: Follow the correct order
- Rule 6: Ask yourself if it sparks joy
Rule 1: Commit yourself to tidying up
Who do we need commitments from to achieve the goal of “tidying up” our technical environments? We need to identify the commitments needed from our system technical owners, business owners, budget owners and leadership.
Rule 2: Imagine your ideal lifestyle
What do we want our technical environment to look like? Is a commitment to remove 100% of unpatchable OSes from our environment realistic? Probably not.
Maybe a commitment to remove a specific operating system or replace/retire a smaller percentage (30%?) of the technical debt in the next 12 to 24 months is more achievable. It will be easier to commit to smaller chunks and make incremental progress toward a greater goal than to fight the uphill battle to get commitment on an “all or nothing” approach.
Rule 3: Finish discarding first
What does “discard first” look like in a technical environment? We rarely throw anything away.
The biggest obstacles to discarding old systems are data retention and particularly data archival. Even if a system is no longer used for daily processes, the data may still be required for legal, regulatory or even “my pet project” reasons. We need effective data archival strategies and solutions that allow us to decommission the legacy system.
How do we stop the problem of legacy data from getting worse? We need to start thinking about data archival as part of a full lifecycle plan for a new application, and for applications inherited through acquisition.
Data Archival and System Retirement Plan should be required chapters in a support plan for a new or inherited system. I know of a small college president refusing to break ground for any new campus building unless both the construction cost and the endowment fund to care for the building were fully funded. This delayed the groundbreaking for some buildings, but this approach guaranteed that the college did not take on unmanageable debt during an eager building phase.
While we build business cases and support models to support new applications, we rarely define the useful lifecycle for an application or include the costs for data archival and system retirement with the business case.
Rule 4: Tidy by category, not by location
In IT, tidying by category might mean looking at all instances of the same operating system and tackling them in one project (e.g., all Windows 2003 OS systems). Alternatively, we could look at all business applications performing the same process/capability. This is similar to an application rationalization approach to portfolio management and provides a disciplined approach to organizing the tidying efforts.
Rule 5: Follow the correct order
and
Rule 6: Ask if it sparks joy
The KonMari Method focuses a great deal on completing your tidying/cleanup once and then maintaining it consistently going forward. This is critical so that you are not introducing “junk” into your home that does not “spark joy.” Most of us have so much technical debt that it may be hard for us to envision completing the tidying process in a timely manner. We should think of unpatchable systems as a marathon and not a sprint. Mark Twain reportedly once said, “Continuous improvement is better than delayed perfection.” We should take the same mindset.
Are data centers similar to our garages or storage areas?
If we relate organizing our garages or personal storage areas to this problem of unpatchable systems, there are some takeaways from this .
Start by setting a goal to clean and separate everything into categories of must keep, want/should donate, sell or throw away. In IT, that includes an up-to-date inventory of our systems that captures the plan for each asset (e.g., keep, consolidate/merge or retire).
There is no perfect approach to doing this, but we all need an approach that works for our organization, can be sustained, and can be leveraged as a data point for strategy planning and budgetary processes.
While the article advises to “give yourself no more than one weekend and take items to the donation center before close of business that day,” our inventory activities may take longer than one weekend.
The recommendations to time-box this activity, confirm our decision process and timeline, and create a way to easily see what we have (e.g., CMDB or application inventory dashboard) can be helpful to get us started.
Once we know what we have, then we can move to planning, gaining support and executing our plans. Several of the projects suggested in the article have to do with effective storage solutions. Have we considered alternative storage solutions for our legacy data? Does it have to be available real-time (e.g., sports equipment, car care products) or could it be moved off-site or archived to be available when needed less frequently (e.g., garage ceiling track storage). Can we update our storage technologies and store more in a smaller (or cheaper) way? We should review this every few years.
All of the projects in this garage organization article share a key concept with the KonMari Method—they all aim to put similar items together for best organization and visibility—or Rule 4 from Marie Kondo: tidy by category, not by location.
We can’t do this alone
Finally, what if we approach this like a community project? The University of Nebraska-Lincoln has an with “Tips for Organizing a Successful Neighborhood Cleanup.”
Two areas that stand out for our problem statement are stakeholder engagement and information research.
“Forming a neighborhood cleanup committee is a great way to get things done efficiently and build ownership at the same time,” the article states. Are we leaving our infrastructure teams to tackle this problem alone or have we formed the right community of application owners, IT support, business leaders and management support? Who has pride in the current solution or would take pride in this project and should be engaged to support the effort? Who will be a dissenter or may try to sabotage our efforts? You may need to create a stakeholder communication plan and build support for your project. Who will organize the project tasks into a detailed plan? Have you accounted for someone to coordinate the activities or are you trying to handle it within your current workload and current staff?
“Research your ‘cleanup area’ to get an idea of the support you will need,” the article offers. Do you have enough information about the legacy system to understand how to move forward with your objective (replace, upgrade, protect further or decommission)? Is key documentation missing or have the subject matter experts left the company? Is there an “archaeology” project you need to undertake to confirm what is required? Have there been other similar projects you can look at to understand how this effort might be advanced at your company? Are your application, cybersecurity and infrastructure teams collaborating well enough to undertake this project or does that need attention?
‘If we chase perfection, we can catch excellence’
The famous football coach Vince Lombardi wisely said, “Perfection is not attainable. But if we chase perfection, we can catch excellence.”
Let’s drive toward continuous improvement and reduce our threat landscapes. Let’s clean out our work “houses” and “garages.” It’s time we tackle this problem with fresh eyes and find what “sparks joy.” Along the way, we just might improve our cybersecurity risk and vulnerability posture at the same time. •
Anita Bateman, CISSP, is an IT executive with experience in the technology, utilities, oil and gas, and automotive industries and is a past contributor to InfoSecurity Professional.