Today's IT organizations face three bad choices as it relates to change:
Update blindly and hope for the best — making out-of-band changes and praying to whichever divine spirits look upon IT for a positive outcome.
Defer updates and hope for the best — again, calling on the same divine spirits for help and hope in allowing continued stability, security and performance.
Pass changes through extensive QA cycles — fully testing every change to ensure it doesn't wreak havoc on production systems.
Weekly change review board meetings can run all day and beyond — debating which of the three options to apply for each of hundreds of proposed changes.
And this is why change is such a dirty word; change is expensive!
The reality is that the pace and frequency of change is not letting up. As systems become more complex — drawing on various third party and in-house components — the sources of change are increasing. Add to this the popularity of Agile development techniques and you get a sense of the plight of enterprise IT.
IT is simply unable to consume change rapidly, economically or predictably.
The answer is intelligently automating change, which begins with the system model. This system model defines the exact profile of a deployed system — the components, versions, dependencies, etc. — providing a basis for delivering surgical, predictable updates to deployed systems.
With a system model in place, patching and updates become far less traumatic. It enables four principles that are essential to intelligent system automation:
Update only what is necessary — creation of a system model allows you to specifically match patches and other updates to deployed systems. This allows you to update only the systems that require changes — rather than blindly implementing wholesale updates because of unknown system inventories. This also prevents the equally pernicious problem of not updating all systems that need it.
Update incrementally — a deep and detailed understanding of the system profile allows you to implement more granular patches and updates. Why risk redundantly updating an entire platform or component when only one file changed?
Don't repeat yourself — by defining systems hierarchically, where variants are derived from common platform components, you can patch and update the base platform once and cascade the changes across all system variants.
Optimize system definitions — a rich understanding of dependencies allows you to reduce update volume by excluding unnecessary platform elements when systems are constructed. You may start by shedding the most obviously unnecessary components or optimize for true thin provisioning with JeOS.
The addition of system version control provides additional update benefits:
Rollback when necessary — a complete system version history allows you to easily revert to any previous version for restoration and troubleshooting. Of course, the entire practice of intelligent system automation makes system outages less frequent, but when they do occur, system version control provides a reliable safety net. This also reduces testing burdens by ensuring that changes are reversible. Today, many IT organizations test extensively around change cycles because the changes cannot be reversed predictably.
Test as a unit — when testing is necessary, system version control allows the entire system definition to be tested as a unit. This improves the efficiency of test cycles by eliminating the possibility of configuration drift between dev, test and production phases and virtually eliminating the risk of deployment failures.
Troubleshoot latent system conflicts — sometimes system conflicts don't appear until well after the offending update is made, creating a murky chain of causality where conflicting updates are sandwiched between innocuous updates. A versioning foundation allows you to corner the culprit change and run it down like a rabbit by reproducing and testing past system versions until you home in on the offending change. In the source code world, this is referred to as “bisection”; traditionally, it hasn't been possible in the world of operations.