Musings on IT, data management, whitewater rafting, and backpacking

Saturday, January 16, 2010

Simple Change Management system pays big dividends

A few years ago, we implemented a simple change management system within our IT support group. We had seen problems with mismanaged changes, but we didn't need some elaborate expensive scheme complete with a Change Control Board as advocated by Big IT schemes.

How does it work?

Wednesday, January 13, 2010

Focus on the big picture

One of the new fallacies of the Googleized Intertubes is that anyone can become an expert on a topic with enough searching and reading web pages and listening to podcasts and watching videos.

Saturday, January 9, 2010

"Call and response" on the river

I've taught hundreds of boats full of whitewater paddlers how to paddle. The basic paddle commands for Class III whitewater are:

Massive Disk Failure, Part 3

We've had a pretty good relationship over 10 years with the disk drive chassis vendor at the heart of this problem. We thought we had our bases covered with their extended warranty promising "overnight advanced replacement" of failed components.

Not quite. Try 2+ weeks to get the chassis replaced.

The problems cited by the vendor:
  • All of this happened in late December, and many of their staff were on "vacation" (or furloughed, we're not quite sure).
  • They did not have our particular chassis sitting on the shelf ready to ship. They had to build one.
  • After building our chassis, they had to "qualify" the chassis to make sure it worked right.
  • The chassis was ready to ship New Year's Eve. Oops. Chassis actually shipped on the Monday after.
Not much we can do about this. Our vendor let us down. Realistically, all we can do is not buy from them in the future.

Massive Disk Failure, Part 2

Most of the dust has settled, though we are still waiting for root cause analysis from our drive/chassis vendor. They found nothing wrong with the first two drives that failed, but apparently this is a common finding.

Recovering from backup tapes was far more painful than anticipated. Despite weekly backups to tape, several people lost 2-3 weeks of work for various reasons: