Musings on IT, data management, whitewater rafting, and backpacking

Saturday, January 16, 2010

Simple Change Management system pays big dividends

A few years ago, we implemented a simple change management system within our IT support group. We had seen problems with mismanaged changes, but we didn't need some elaborate expensive scheme complete with a Change Control Board as advocated by Big IT schemes.

How does it work?

  1. A system admin fills out a simple Web form with a Change Request
  2. The CR is emailed to all the other IT support people, and to me (the manager)
  3. I approve or disapprove the CR in return email
  4. Our webmaster adds approved CRs to our Change Schedule web site, which acts as both a forward look at upcoming changes, and a record of what we have changed.
What makes this scheme work is the rules that surround it:
  • All major changes to multi-user systems must have approved CRs before sys admins can make non-emergency changes.
  • Changes made without approved CRs must be undone immediately.
  • CRs must be sent about one week before the proposed change.
  • We have blackout weeks immediately before and during business-related crunch times. These total about 4 weeks per year.
  • Changes must be made on Tuesdays unless you have a really good reason for another day.
  • If you are making the change, you can't leave for vacation or other planned absence for a few days afterward.
  • The form asks for detailed information. If I don't get all that information, I'll deny the CR.
Why Tuesdays?

Mondays are bad because people need to get some work done after the weekend, and Mondays are often holidays.

Thursdays and Fridays are bad because often we see problems 1-3 days after a change is made (e.g. server crash), and we want sys admins and vendor support staff available. Most of our support contracts cover 8 am to 5 pm Monday-Friday.

That leaves Tuesdays or Wednesdays. We chose Tuesdays.

What's a Major Change?


A major change is any change that could affect two or more people. Over time, we refined what is a "major change" and what is "business as usual".

Major change examples:
  • Any server operating system or application update or install
  • Any firewall change
  • Any network change
  • Any patches or software that we push out to users, or advise users to install.
What fields are in the form?
  • Change title, including system name
  • Requester's name and email address
  • Names or descriptions of the systems affected
  • Planned date/time of change start
  • Planned date/time of change end
  • Short description of change for users
  • Long description of change for system admins
  • High/Medium/Low rating for User Impact
  • Backout plan, i.e. what will you do if the change doesn't work right?
  • Backout time limit, i.e. when will you "pull the plug" on the change and go back to what worked?
  • Benefits of change
  • Risks of change (I've rejected many CRs for glossing over this part)
  • Communication plan: Dates & times to send warning emails to users
  • Communication plan: Status update email after success/failure with brief explanation.
Believe it or not, our simple Change Request system is significantly better than virtually all of our Enterprise IT systems. I've seen no communication before major changes; incomplete descriptions of the changes with major impacts; changes dragged on for hours or days as they struggle to get to the goal rather than pull the plug and reschedule; and changes made without approval by anyone besides the responsible sys admin.

When we first implemented our CR system, our sys admins complained bitterly. Now, everyone uses it and expects everyone else to use it. A couple of times someone has forgotten, and that has caused a lot of problems.

As we've gained more experience with the system and trust in the sys admins, I've permitted some flexibility in notices, change days, etc., after getting good justifications.

How would I improve our system?
  • Automate the approval and posting of approved change requests. For example, when the CR email goes out, I should go to another web form, authenticate, and approve/disapprove/comment on the CR. If the CR is approved, the CR should automatically be added to the Change Schedule.
  • Automate the sending of warning emails to users from inside the CR system. The sys admin would compose warning emails inside the CR, and select dates/times to send the warnings. The CR system would automatically send the warnings. Sometimes sys admins forget to send warnings, or they are out sick.
  • Consolidate change warnings to users. Some Tuesdays we might have 6 pending changes. Consolidate warnings of those changes into no more than one email per day on the days before the change.
These changes would require significant programming to extend our current dirt simple system. We will explore using the Change Management tools built-in to our WebHelpDesk system instead. BTW, we really like WebHelpDesk. More on that later.

Simple Change Management works for us. Can't imagine not using a Change Management system now.

2 comments:

  1. What do you think about OTRS with the ITSM modules installed?

    ReplyDelete
  2. Anonymous,

    We did not do a comprehensive survey of help desk or change management systems. I wanted something simple and easy to implement. Most open source systems are neither, since we don't have a lot of wizardry in our shop. We would have used WebHelpDesk's built-in change request system if we had that at the time.

    ReplyDelete