Rocks Sand Water

500 Miles on the PCT – Lightening Up

2012-01-27T22:14:00.000-08:00

In August 2003, I tried to hike 500 miles of the Pacific Crest Trail through the High Sierra in California – in 30 days.

Here's how I reduced my pack weight ...

The single most important piece of equipment I bought never went on the trail with me: an inexpensive digital scale! Weighing everything to the nearest gram is quite important. Manufacturers weights are averages (or exaggerations), and after you modify your equipment, you need to weight it again.

Modify your equipment – every gram counts! There's no gear police that will swoop down after you remove the straps, logos, cords, belts, and other doodads that you don't need. You might even add something, but think long and hard. Be brave and cut that hole that makes something work better. Weight and function are far more important than appearance.

On my first weekend trip, I used old equipment (some very old). That gear worked OK, but was over 25 pounds base weight.

Piece by piece I bought lighter versions of my old gear.

After a couple more weekend trips trying the new gear, I was still over my target base weight, and several items didn't work well.

Time to re-think everything from the ground up. Back to Ray Jardine's books and the Internet.

Lightweight backpacking requires new skills, not just lighter equipment and book learning, so I tried new equipment on my training trips.

On one trip, I used a poncho as a tarp, which failed miserably – too small for me. I switched to a Tarptent Virga with sewn in floor, about the same weight as a tarp plus ground sheet plus poles (I don't use hiking poles). Unfortunately, on a training trip with that tent, a tick snuck under the unsealed entrance. I got bit and infected with staph, and had to end the trip early. I used that tent for the PCT, since in the High Sierra, ticks are not a big problem.

On the same trip, my gas cartridge seal leaked, and I soon had an empty cartridge, a smelly pack, and no way to cook. I switched to an Esbit Pocket Stove with Esbit tablets, slightly modified. Much simpler, lighter, and more reliable.

All the training trips taught me which equipment worked, didn't work, or needed modification. I made many other equipment changes, large and small, based on those experiences.

For my final training trip, I used exactly the equipment I planned to use for the PCT – I didn't want any surprises in the High Sierra.

My base weight was higher than I wanted, but I didn't have enough time to reduce it any further.

Final Equipment List – Weights are from my digital scale.

Carried
Ounces	Description
31.2	Granite Gear Vapor Trail pack, after trimming
26.0	Western Mountaineering Megalite 30°F sleeping bag
8.2	ThermaRest Z-rest pad, after trimming
24.9	Tarptent Virga with floor
3.5	Esbit stove + windscreen
0.7	Lighter
5.5	Snow Peak Ti Solo cook set
0.6	Lexan spoon
0.5	Scrub pad, mesh sack
1.4	Soap, 1 oz in squeeze bottle
1.4	Antiseptic hand gel, 1 oz in bottle
3.4	Water bottle 2x1.5L
3.2	AquaMira water treatment
37.0	Bearikade Expedition canister
1.1	Black Diamond headlamp
0.3	Spare battery, fits headlamp and camera
4.5	Sunscreen
3.0	Bug repellent (3M Ultrathon)
0.6	Toothbrush
1.7	Toilet paper in plastic bag
2.0	Trowel
1.4	Pack towel
9.6	First aid kit
3.8	Whistle, mirror, tape, matches, firestarter
1.1	Chouinard sewing kit
4.0	Stuff sacks
4.1	Maps
1.6	Drivers license, credit cards, cash
5.4	Olympus Stylus Epic film camera
2.0	Film, 2 rolls
0.3	Comb
2.5	Cord to hang food
2.3	Trash compactor bag
3.5	Journal, pencil
9.8	Silnylon poncho
8.7	REI fleece top
7.1	REI polypro tights
5.6	Montane Lightspeed jacket
3.4	Montane Featherlite pants
5.0	Socks, 2 pair
1.3	Clear glasses
1.3	Glasses case
0.8	Reading glasses
1.3	Glasses case
1.4	Fleece gloves
1.5	Fleece hat
1.1	Bandanna

250.6	Total Ounces Carried
15.7	Total Pounds Carried

Worn
4.0	Tilley hat
1.3	Sunglasses
8.5	REI MTS T-shirt
1.9	Suunto Vector watch
7.1	Patagonia Baggies shorts
2.5	Socks
4.0	Orthotics
29.8	Asics Gel-Eagle Trail II shoes

59.1	Total Ounces Worn
3.7	Total Pounds Worn

Since the PCT trip, I've reduced my base weight by a couple of pounds, based on more experience.

Next – the trip

500 miles on the PCT – Training Trips

2012-01-26T22:41:00.000-08:00

In August 2003, I tried to hike 500 miles of the Pacific Crest Trail through the High Sierra in California – in 30 days.

The best way to get into shape for backpacking is ... to go backpacking. I wanted to backpack 500 miles in several shorter training trips before the big trip.

Here's what I did ...

I needed to take those trips in the fall, winter, and spring. That ruled out the High Sierra, and left me with few long-distance trips in central California.

I started with one weekend trip using 20 year old equipment lying around. Too heavy, but I'll cover gear later.

After a flurry of buying the wrong stuff, I did a couple more weekend trips.

Then I mapped out a 100 mile backpack loop through the Santa Cruz mountains, and long loops through Henry W. Coe State Park east of San Jose. So I planned five 100 mile trips using those loops.

On my first long trip I had serious equipment problems, including a fuel cartridge leak. I ended that trip several days early. And I rethought my equipment needs from the ground up.

My second long trip went well. Everything worked OK hiking 9-15 miles per day. I did have one scary off-trail adventure, and learned to always stay on well-traveled trails.

My third trip, in March, brought hail and sub-freezing temperatures on the second day – unusual for coastal California. My gear couldn't keep me warm enough. After one very cold, sleepless night, I ended that trip several days early.

For my fourth trip, in April, I planned to hike 16-20 mile days. After two days, I got a staph-infected tick bite because my tent entrance didn't seal. I had to end that trip early to get major antibiotics.

For my last training trip in May, again I planned to hike 16-20 mile days. I caught a cold just before the trip started – and after two days, I hiked out before I collapsed.

I had finished only one out of five training trips, and backpacked less than 200 miles. I was way behind schedule, but I had no more time available before August.

Not good.

Next – Lightening up!

500 miles on the PCT – Getting Into Shape

2012-01-25T22:49:00.000-08:00

In August 2003, I tried to hike 500 miles of the Pacific Crest Trail through the High Sierra in California – in 30 days.

Here's how I got into shape for the hike.

I needed to get my hiking mileage up to 20 miles per day. I used to hike as fast as I could until I got tired, rest for a while and drink a little water, repeat until done.

I hit the wall at 13 miles per day – dead tired, achy, sore, lots of foot blisters – even with light, ankle-high boots. The aches and pains got so bad I was taking Aleve before and after hikes!

I re-read Ray Jardine's books, read the latest edition of Colin Fletcher's The Complete Walker, read Fixing Your Feet, scoured the Internet, and started experimenting on hikes.

I learned some important lessons:

Slow down to 2 mph average. I deliberately slowed my hiking pace to average 2 miles per hour, including breaks. You need to hike more hours per day, not faster.

Take 10 minute breaks every hour. Hike 50 minutes, rest 10 minutes, with a pace of 2.4 mph while hiking. I'm not rigid about stopping on time – I start looking for a nice place to stop a few minutes before, and sometimes keep looking a few minutes after.

At every break, take off your shoes and socks, and let your feet air as long as possible. Heat and moisture cause blisters, so cooling and drying them helps prevent blisters.

Carry two pairs of socks, and put on the drier pair of socks at each break. For example, after a stream crossing, the socks in my pack still might be pretty wet, so I'll keep wearing the old socks, which had a few minutes to dry.

Kill athlete's foot with Vicks VapoRub. The fungus causes your feet to peel and blister more easily. I applied a thin layer of VapoRub daily, to all of the infected areas. In a few weeks, my fungus was gone, plus my feet smelled better.

Carry and use a sit pad, to encourage sitting at breaks. I used the part I trimmed off my Z-Rest for day hikes. For backpacking, use your sleeping pad, or skip the pad entirely.

During breaks, keep your feet up – preferably higher than your butt. This reduces fluid buildup in your feet and legs. Avoid sitting chair-like on logs or rocks.

Stop taking painkillers, before or after hiking. Prostaglandins signal muscles to get stronger. I needed to tough it out, and not take Aleve, Aspirin, or Ibuprofen (antiprostaglandins) for the aches and pains. Within weeks, I felt stronger!

Drink plenty of water. At each break I drink 1-2 cups of water, from 1.5 liter Trader Joe's water bottles – light, cheap, and simple. Some people like fancy hydration systems, not me.

Switch from boots to trail running shoes. Following Jardine's advice, I tried street running shoes, but my feet got beat up worse than boots.

Avoid Gore-Tex lined shoes. Gore-Tex keeps moisture and heat in, and doesn't keep water from sloshing over the top.

Buy bigger shoes. Your feet will swell on long hikes. I went up 1/2 size, and that worked fine.

Tie your shoes loosely, so you can slip them on and off. I stop lacing two holes from the top, and don't tighten the laces much. Loose shoes have many advantages I won't describe here. I haven't had any trouble hiking in loose shoes, but they take a while to get used to.

Hiking poles don't work for me. I tried double hiking poles, I tried single hiking sticks. They got in the way and seemed like extra weight to carry. I never saw any of the miracle benefits others find in hiking poles.

Soon I broke through the 13 mile wall. In a few months, a 20 mile day was long and tiring, but not painful. 10 miles before lunch felt like a warmup hike. And I almost never got blisters.

I dedicated Fridays to long solo hikes, plus shorter hikes on Sundays with my wife. My work week was Monday through Thursday. Hiking is not the same as backpacking, but if you can't do the miles hiking, you can't do them backpacking.

I started adding weight to my day pack, using several 1.5 liter water bottles. Later, I started day hiking with a full backpack, using water bottles in place of food and other gear.

I gradually worked up to my target weight of 25 pounds, and my target distance of 20 miles per day.

Next – backpacking training trips.

500 miles on the PCT – A Crazy Idea

2012-01-25T22:38:00.000-08:00

In August 2003, I tried to hike 500 miles of the Pacific Crest Trail through the High Sierra in California – in 30 days.

Why?

First Try

In 1976, I was ready to drop out of graduate school.

But I needed a new goal.

I started backpacking as a teenager in the Boy Scouts, but it wasn't much fun with the primitive equipment and techniques. In high school and college, I'd gone backpacking many times with friends (and better equipment), and we all had a great time.

The PCT route had recently been established (but not finished), the Wilderness Press PCT guidebooks made the trip seem possible, and Colin Fletcher's book The Thousand Mile Summer provided the inspiration.

My new goal became thru-hiking the California section of the PCT, alone, in six months. That's about 1,700 miles in 180 days, averaging 9.4 miles per day (stop laughing).

After a frantic couple of months preparation, my best friend dropped me off at the Mexican border on April 1. The border fence was a few strands of rusty barbed wire back then.

I dropped out a few days later at Mt. Laguna, with blisters covering about one-third of each foot. My 65 pound pack, lack of conditioning, and isolation all contributed to my failure. Hitchhiking home, I nearly died from hypothermia in the back of a pickup truck.

Helping Others

In 1977, four college friends started thru-hiking the entire PCT. I helped them plan and pack, met them along the way, and generally encouraged their trip. One friend made it all the way. Everyone had a life-changing experience.

Second Try

In 1980, my best friend and I tried to thru-hike the entire PCT. To make a long story short, I dropped out and returned twice in the first month due to injuries and illness. I finally dropped out for good at Weldon. My body rebelled at both the food and exertion (50-65 pound packs again), and my mind couldn't overcome those obstacles. My friend dropped out a month later.

Twenty years of dreaming, and then ...

In 2002, I was approaching one of those Big Birthdays, and figured I'd better hike the PCT sooner rather than later.

I was inspired by Ray Jardine's books on hiking the PCT using lightweight gear and techniques. This time for sure! I needed to get into shape, and reduce my pack weight.

I chose to thru-hike "the best of the PCT", 500 miles southbound from Donner Pass to Walker Pass. I wanted to complete that hike in 30 days, in August 2003.

Next – getting into shape.

Career change

2011-10-19T22:36:00.000-07:00

After 35 years in IT Support and Data Management (mostly IT support), I'm moving completely out of IT Support and into Data Management full-time.

No more obsessing over speeds and feeds of handhelds, desktops, servers, networks and data centers. No more struggling to implement insane support and security policies. No more late night and weekend support calls.

Of course, Data Management has a new set of challenges and headaches.

I asked for this change, but it's still a shock.

This blog will change, too.

Stay tuned.

RIP Steve Jobs 1955-2011

2011-10-05T18:20:00.000-07:00

I am very sad. More later.

Last Comcast Update

2011-08-19T22:29:00.000-07:00

Comcast chose not to bid on our Gigabit Ethernet RFP after 16 months of negotiations.

So we're getting AT&T Gigaman Metro Ethernet instead.

In my first two Comcast blog posts, I was surprised that Comcast offered both high speeds and very competitive prices.

We started discussions with Comcast in early 2010, which continued through most of the process. Due to several SNAFUs on our side, we didn't get an RFP out until this spring. We got several proposals with order of magnitude price differences.

Much to our surprise, Comcast chose not to bid, so we signed a multi-year contract with AT&T.

Since then, communication with AT&T national has been difficult, while the local installers have been great. In fact, we've got the same installer for our Gigaman service who installed our DS-3 over 7 years ago!

Our Gigaman circuit gets turned on Monday, we'll see how it goes.

Our Gigaman circuit is single-mode passive optical fiber from our network interface to the AT&T Central Office, with no electric power required at our end. This circuit should be more reliable than our very reliable DS-3 service, which has three AT&T wall-mounted boxes which require customer-supplied electric power.

I compared network outages on our DS-3 to our "backup" dual T-1s. Our DS-3 has been over 10 times more reliable than the T-1s.

We're still using Comcast for our visitor's network. We just had Comcast move the termination from our old telecom room to our new main telecom room.

Still no demand for Comcast TV or phone service, but that might change in a few months.

New server room first impressions

2011-08-12T12:40:00.000-07:00

We're moving into our new server room (data center is a dirty word now). I've mentioned many of the design elements in previous posts.

Here are my first impressions and a few lessons learned.

Major design elements:

480 square feet
60 KW IT power feed
Concrete slab floor with 6 inch high concrete pedestals under the server cabinets
Six sealed server cabinets with blanking panels and chimneys for heat exhaust
Room for six more server cabinets
White cabinets, white walls, and soon, white flooring
Sound absorbing foam on two walls
Two HVAC units on the roof; each can handle 80% of maximum load
Aggressive air-side economization
Supply temperature floats between 70 F and 86 F
No humidity controls
Overhead telecom ladder racks
Overhead power busses
208 VAC single phase power for most IT loads
Rack-mount UPSes with generator backup for IT loads
Generator backup for HVAC

We have about 5 KW of active IT load now (as measured on the buss), with more servers moving in over the next few weeks.

So far, I'm happy with this design.

Some compromises are causing minor issues:

– Our off-the-shelf HVAC units do not have variable speed blowers (VFDs). Each unit blows 5,000 CFM, and is on or off. 5 KW IT load needs about 500-600 CFM, so we are overpressurized. The room doors tend to blow outward, but not dangerously so. We can reduce the overpressurization simply by removing some blanking panels, but we need to be careful not to mix cool and warm air. Custom HVAC units with VFDs were several times more expensive.

– Our contractor surprised us by painting the existing stained and scarred concrete floor (the old floor was fine with me). Now, we're vacuuming up small snowdrifts of paint chips daily, to keep them out of the servers. I'm getting an estimate for floating vinyl flooring next week. Next time I would spec vinyl flooring.

– We have a 75 KW transformer in the server room, in spite of my spec for "no high EMF sources in the server room". We moved the transformer to a corner just inside the door to minimize the EMF impact on the rest of the room. Transformer magnetic fields fall off as the cube of the distance, so this shouldn't be a big problem. The transformer generates some heat, too. Our main problem is keeping people from parking sensitive equipment or media on the handy flat topped surface.

– The cabinet chimneys are fitted into an overhead exhaust duct, instead of a dropped-ceiling plenum. This design, chosen by the HVAC subcontractor, took a lot of custom sheet metal work and minor modifications to the chimney mounts. Adding more cabinets in the future will require more custom work.

Other issues that came up:

– Clear communication with the HVAC subcontractor was challenging. My design goals were so different from what they had installed elsewhere, that they kept making incorrect assumptions. However, the back-and-forth was useful. The lower temperature bound of 70 F is much higher than my original spec of 60 F, but recirculating IT-generated heat is free. 70 F gives us plenty of margin above the highest dew point we've recorded in the past year of 62 F. We also lowered my upper bound spec from 90 F to 86 F. The cooling lag when the compressors kick in mean we could peak at about 90 F.

– The HVAC subcontractor ran supply ductwork into the "cold" aisle, even though my design doesn't require that. Could have just dumped the cool air through a large screen in the wall. We don't need precise distribution of cool air since we have almost no warm and cool air mixing.

– Never again will I say something like "how hard can it be to install cabinets?" Installing 6 cabinets took 2-3 of us 3 full days, not including connecting chimneys to the overhead exhaust duct. And then we spent several days off-and-on fixing problems with the floor bolts and brackets. Next time, I'll pay for professional installation.

Backpacking food

2011-07-12T23:07:00.000-07:00

I've tried all kinds of systems for planning, preparing, and eating backpacking food. At one point, I could tell you how many calories, how much fat, how much protein, yadda, yadda, for every part of my backpacking diet.

Useless. Totally useless.

Old Army saying: Doesn't matter how many calories you give a man if he won't eat it.

Absolute truth. Start there.

Here's what worked for me, and what I'd like to try for future trips.

On my last long trip, my menu looked like this:

Breakfast
Grapenuts	4	oz
Milkman 1/2 pouch	2	oz
Raisins	0.8	oz
Vitamins	3	each

Snacks
Larabars	4	each

Lunch
Baked cheese	2	oz
Hardtack	2	oz

Dinner #1 – Rice Pilaf
Freeze-dried dinner	1	each
Olive oil (1.5 TB)	0.5	oz

Dinner #2 – Couscous
Couscous	4	oz
Pine nuts	1	oz
Parmesan cheese	1	oz
Sundried tomatoes	1	oz

Dessert
Coconut carob bar	1	oz

Average 27.2 oz per day

A little light, but I enjoy all these foods, and gorge when I get to a town or restaurant.

If I felt the need to bulk up, I would add more snacks and a bigger lunch.

Do not add more Fat to your diet thinking calorie density, e.g. adding more oil to dinners. Your body will reject it after a week or so.

About the meals and ingredients:

If I have the fuel and the time, I love making breakfast with hot water. Let the raisins and Grapenuts soak for a little while.

Milkman was off the market for a few years. Just a few months ago Packitgourmet.com made Milkman available again.

Fortunately, Nestle Nido is a great substitute, and higher in calories since it's full-fat. Expensive online, hard to find in regular stores, try the Hispanic foods section. Also try Walmart, or larger Hispanic grocery stores.

Larabars are perfectly packaged for my long distance hiking needs. I eat one every other hourly rest stop, sometimes every rest stop. They taste good, come in a wide variety of flavors, keep well, and they are real food – a rare combination! Much better than the chemical soup energy bar products on the market.

Specialty Cheese Company's "Just the Cheese" brand baked cheese is wonderful. Crunchy cheese goodness, high in calories, well packaged, and no funky chemicals.

Hardtack is from my own recipe. You'll find some commercial alternatives there if you aren't into simple baking.

Looks like Mountain House has discontinued Rice Pilaf, the only freeze-dried dinner I've ever liked. I'll try their Pasta Primavera, or reach back into my recipe archives for a new Dinner #1. I like to have a little variety at dinner.

The Couscous dinner is my new favorite. One great feature is no wasted water – all the water you use for cooking you eat with the food.

The Dessert is available in the bulk bins from some local natural foods stores, but I could never find the original manufacturer. Like a Mounds bar made with carob and without sugar. I rarely eat dessert at home, but highly value that little snack just before bed time on the trail. Helps me stay warmer at night.

Backpacking Foods I want to try:

Bear Valley Pemmican or Meal Packs have been around for decades. I ate too many of them when I was younger, but recently discovered that they taste good again. Like Larabars, simple real food, well packaged, variety of flavors. I want to try eating one of these for breakfast-while-hiking.

I've become addicted to tea. At home, I was fussy about what I drank, until some diet changes forced me to try something new. Much to my own surprise, I love Lipton Unsweetened Instant Iced Tea Mix, with lots of Nestle Nido, made hot (not cold).

Just discovered PackitGourmet.com. Lots of new, backpacker friendly items to try!

My Diet Restrictions

Everyone has dietary preferences or requirements.

I'm a lacto-ovo vegetarian by strong preference for health, ethical, and ecological impact reasons.

Sugar, artificial sweeteners (especially Aspartame), and excess salt make me sick, so I avoid those, too.

I've eaten this way for over 30 years, while hiking thousands of miles, and guiding whitewater rafts on many challenging multi-day trips. You don't need to eat a standard American diet to be healthy and vigorous.

Hike your own hike, live your own life.

Electric car "range anxiety" workaround

2011-06-16T15:26:00.000-07:00

One solution to electric car range anxiety: When you buy an electric car, you get a life-of-the-car discount on renting a regular car, including parking and charging privileges.

If I need to take a long trip, I drive the electric car to the rental car company, move my stuff, plug-in my electric car, and drive away in the regular car. Reverse the steps on return.

Enterprise might be a good choice, since they have many locations.

Simple, and not too much hassle.

Nissan? Are you listening?

Two missing storage performance metrics

2011-06-09T08:17:00.000-07:00

I would really like to see storage system vendors or reviewers release consistent performance metrics for:

Worst case RAID rebuild times

Worst case storage fill times

Then I can choose my Maximum Storage Pain Threshold.

Recently I've been through total storage system failure, or right at the edge of failure. What I didn't know, and couldn't plan for, was how long recovery would take.

Scenario #1 – Full RAID rebuild:

A RAID 5 system suffers single disk failure, or a RAID 6 system suffers double disk failures.
Rebuild starts to hot spare(s) or swapped disks.
How long will that rebuild take?

This number is almost impossible for customers to compute from component specifications. Each system has unique variables including:

RAID controller performance
Onboard RAID versus HBA RAID versus Software RAID
Alternative data protection architectures like ZFS, WAFL, BeyondRAID, ...
Storage system interfaces including SATA, SAS, FC, iSCSI, ...
Disk interfaces including SATA, SAS, FC, ...
Disk performance specs like total capacity, rotational speed, latency, seek times, I/O transfer rates, cache size, ...

Given all these variables, we probably need a scheme like SPEC with standard workloads, and the vendors or reviewers report the exact hardware and software used for each test. Testing like that still has problems, but it's better than nothing, which is what we have now.

Scenario #2 – Full restore:

A completely full RAID 5 system suffers double disk failures, or a RAID 6 system suffers triple disk failures.
Rebuild is impossible, so we swap disks and reformat
We start restoring from backups – tape, another server, another storage array, the Cloud. For the purpose of this exercise, assume an infinitely fast backup source.
How long will that restore take?

In addition to all the factors in Scenario #1, now we are stressing total system throughput. In this scenario, using the same SPEC-like model and computing maximum theoretical write speed might be sufficient.

What's my Maximum Storage Pain Threshold?

Short definition: It's the amount of time the storage system can be down before my job is threatened.

Longer, more cynical definition: Starting with a thorough analysis of business needs, blah, blah, blah, manipulate the numbers to match my budget and the amount of time the storage system can be down before my job is threatened.

MSPT will be a combination of times I've discussed before:

Corruption Recognition
Recovery Initiation
Data Recovery Time

MSPT will vary depending on the criticality of the data.

Adventures with World3 1976-1978

2011-05-30T22:36:00.000-07:00

From late 1976 to early 1978, I ran many simulations using the World3 model. World3 was used for the famous book The Limits To Growth.

I was puzzled by my results for many years, until I had an Aha! moment.

In the late 1970s, I had a passion for environmental politics, a full-time data processing job with only half-time of work, and nearly unlimited access to an IBM 360/50 mainframe. The Limits To Growth, based on the World3 model, had caused a major uproar for years after publication.

Our mainframe had the DYNAMO simulation language used by World3, so I keypunched the source code from Dynamics of Growth in a Finite World, and started playing. I had very little background in modeling or computer simulation.

The Limits To Growth showed simple and convincing graphs going out a few decades, which supported the book's main premise that the world was headed toward a dramatic crash in less than 20 years.

One of my first puzzles came from running World3 out a few more decades. Many of the graphs went nuts just after the cutoff dates used in the book. By "nuts", I mean wild fluctuations, sometimes into implausible values, like negative values for quantities that could never be negative. So I restricted my experiments to the same time frames as the book, and started varying other parameters.

I found that very small changes in the starting values could cause dramatic changes in the outputs. I had an sense at the time that this wasn't right, but since I didn't know much about modeling, I thought I was just ignorant.

In early 1978, I changed jobs, and no longer had access to DYNAMO, World3, or free mainframe time. Soon, I had mostly forgotten my experiments.

Many years later, while learning about Chaos theory, I had an Aha! moment.

I wasn't stupid.

World3 was a chaotic system driven by simple non-linear equations.

Facebook's Prineville Data Center

2011-04-20T09:38:00.000-07:00

Data Center Knowledge posted two video tours of Facebook's new data center in Prineville, OR. This data center uses Facebook's Open Compute Project designs. The videos describe many features not found in the Open Compute Project documents.

Here are the direct links on blip.tv:

Inside Facebook's Server Rooms
http://blip.tv/play/g7dBgrSxRgA

Facebook's 'Penthouse' Cooling System
http://blip.tv/play/g7dBgrTabAA

Observations and comments on features not found in the Open Compute Project documents:

No lights in the warm aisle, since all work is done from the cold aisle.

Lights on motion sensors everywhere else. Others have reported that all lighting uses LEDs.

Custom reactor power panel, described in more detail here, handles short-circuits, harmonics, and power factor correction.

Some of the server-warmed air is used to warm the office space. What about cool air for the office space in the summer?

65° F to 80° F supply air for the cold aisle. Not as aggressive as some other designs, which use cold aisle temperatures 90° F or higher.

The penthouse fan wall pressurizes the cold aisle to 0.03 inches water column. This pressure takes most of the load off the server fans, so the server fans run very slow. The large VFD fans used in the fan wall should be much more efficient than the small server fans. Some server designs eliminate server fans entirely. I wonder why Facebook choose to keep server fans?

The penthouse fan wall was moving about 340,000 CFM during the tour, with a target of 25-28° F delta-T through the servers. That translates to roughly 2 MW of IT load cooled.

Original design used only expensive air filters, but Facebook temporarily added cheap air filters to mitigate construction dust. They plan to keep the cheap air filters, which should extend the life of the expensive air filters.

Looks like the outside air plus server-warmed air mixture is run through the filters. This seems redundant – the server-warmed air was already filtered. Would have been more efficient to place the filters only in the outside air stream, but maybe Facebook has other reasons for re-filtering server-warmed air.

Facebook uses VESDA smoke detectors in the air supply. Here's a short YouTube video explaining VESDA smoke detectors.

One brief shot showed a cold aisle with no server racks (“triplets” in Facebook nomenclature), but with battery boxes and giant blanking panels in place of the missing triplets.

The use of large plenums, and walls of fans, filters, misters, etc., enables easy maintenance while operating.

The battery boxes can supply 90 seconds of backup power to the servers. Facebook has programmed them run for 45 seconds. Yet the generators need only 15 seconds to startup. Why the overkill on the battery box design?

The engineer claims the mister pumps run all the time, so they use the same amount of power whether the misters are spraying or not. Pumps should use more power if you are actually spraying water, but maybe they are using a recirculating pump design.

Misting water is drawn from on-site wells. The on-site storage tank holds 48 hours of water, with both city water and trucked-in water as backup.

The misting water is run through reverse osmosis to remove minerals, and sterilized to remove bacteria. RO and sterilization can use a lot of power. With a claimed PUE of 1.07, they can't be using much.

Because of the innovative dual AC and DC server power design, the battery boxes consume only battery trickle charge power until AC power is lost. Power sensors switch to battery power at any sign of trouble, similar to standby UPS designs. No double-conversion overhead typically found in large data center online UPS designs.

Facebook has not released many key specifications of their data center design, including:

Reverse osmosis and water sterilization system

Custom reactor power panel

Custom rack mounted network switches

Overall network design

I hope Facebook plans to release more specifications soon.

My 2006 IT forecast

2011-04-19T18:26:00.000-07:00

In March 2006, I was asked to write an IT forecast for upper management, to guide strategic planning. I stumbled across that forecast again today, removed some internal references, and posted it below.

To IT people, and in hindsight, these forecasts seem obvious. But they were not obvious to non-IT upper management at the time.

Consumers will drive most advances in information technology, not businesses or Governments.

Broadly, IT advances have been driven by three waves:

First wave: Government requirements

From the Census of 1890 through the 1970s, most computing advances were driven by Government requirements. Much IT was developed to meet the unique requirements of the Government, then commercialized for use by business.

Second wave: Business requirements

From the 1980s to the early 2000s, we bought commodity computers that were sold worldwide. These computers and other IT were developed to meet the needs of large businesses. This drove down the cost of IT, but made meeting our unique requirements more difficult in other ways, including increasing security problems. The PC revolution brought widely available computers, networks, and software, so much so that we barely remember Government-unique systems, except in DoD.

Third wave: Consumer requirements

Increasingly, consumers are driving IT requirements. A much larger market than for business or governments, we're seeing costs come down substantially, ease of use increasing, but security, reliability, and maintainability dropping. Cell phones, wireless networks, cheap Internet access, digital cameras, iPods, high-speed graphics cards and personal computers designed for video games, GPS receivers, are all developed for consumers, then re-used by business and the Government. The rate of change of consumer technology is much faster than for the previous waves.

Implications for planning:

More than ever, our needs will be met by cobbling together and reusing technologies designed for other purposes. When iPods are used to help identify birds using songs and pictures, blanket policies prohibiting the purchase of “consumer” technologies will hurt our science.

Some of our needs (for reliability, security, ...) will be more difficult to meet, and will require new approaches. For example, rather than building one very reliable data acquisition system connected to one reliable network, we might be forced to collect the data several times with cheap devices, send the data several times over a variety of unreliable networks, and figure out which data stream is the “correct” one. And when one of those devices or networks fails, we'll need to be flexible enough to use the “new improved” model that doesn't work exactly the same.

Central control of IT will be virtually impossible. As long as scientists can acquire very powerful technology with credit cards, or simply buy very cheap technology out of their own pockets, we must lose any illusion of “controlling” our IT environment. For example, restrictions on Internet access are nearly impossible today – I can access the Internet from my office through at least four completely different wireless networks, three of which have no policy covering my use.

Disk storage per dollar and per disk drive will grow dramatically for the foreseeable future – doubling every 18 months or less.

Implications for planning:

We will collect a lot more data a lot cheaper than before. This is a double edged sword – we'll be able to do things we couldn't do before, but we'll be swamped by all the data we can collect. For example, one group has switched from collecting only data from rare, significant events from their worldwide network, to continuous recording, and the research benefits are only starting to be explored. Continuous high-definition video recordings of a variety of natural phenomena are creating whole new research techniques and scientific discoveries.

Our data management problems will double every 18 months, and other parts of the system won't be able to keep up. For example, we should be keeping offsite backups of all our data. Tape drive capacity isn't keeping up, and network capacity isn't keeping up. We will see some very large datasets lost to disaster, accident, or deliberate destruction in the near future.

We'll need to prioritize what data we keep, and what data we protect, because it will be too easy to “keep everything” but protect and use almost nothing.

Maintaining good computer security will not get easier.

Hackers improve attacks as fast or faster than we improve defenses. People are always the weakest security link, and human nature hasn't improved substantially for a million years. New technologies are developed with little thought to computer security, especially as profit margins decline and consumer ease-of-use (see other IT trend) trumps improved technical security.

Implications for planning:

Cooperating with external partners will not get easier, unless we radically rethink how we do either computer security or collaboration.

Resources devoted to computer security will not go down, even as our overall budget shrinks.

Improving security to meet external mandates will have a major impact on budgets, major operational impacts on collaboration and even using computers to do our science.

Scientists will silently subvert computer security policies and technologies in order to get some work done.

Insider attacks will increase as our workforce gets more computer literate. We won't be able to use a security model that trusts “us” but doesn't trust “them”, for arbitrary definitions of “us” and “them”.

Technical security measures will improve, but improving processes, and changing people's computer security habits, will remain major challenges. When mandatory computer security classes reach 40 hours per year...

Raw computer speeds will increase more slowly than in the past. Multiple-core CPUs, and multiple CPUs per computer, will become the norm.

In a few years, desktop computers might contain the equivalent of eight central processors, each one only a little faster than today's fastest chips.

Implications for planning:

Lots of software must be rewritten in order to effectively use multiple CPUs to analyze larger data sets. Old software that's not rewritten will not run substantially faster in the future, and won't be able to handle exploding data set sizes.

Software development, acquisition, and maintenance costs could rise substantially.

Our programmers and scientist/programmers will need training in new software development techniques, and will need new software tools to write, debug, and maintain software running on multiple CPUs.

Our system administrators will need training to effectively run multiple-CPU desktop and server computers.

END OF 2006 FORECASTS

How about the forecasts I didn't make?

I deliberately omitted grid computing, now known as cloud computing, because we had too many internal and external barriers to widespread use. Turns out that was a good call. Except for a few people using consumer clouds like Gmail and Google Docs without support or authorization, we've made almost no use whatever of cloud computing. Most of the barriers are still in place, despite top management pronouncements to “consider clouds first”. And we've added barriers to consumer clouds like Facebook, Twitter, and Dropbox.

More Weather Data!

2011-04-12T20:35:00.000-07:00

Facebook recently announced their Open Compute project. Good stuff for large data centers.

Facebook's design was optimized for the weather in Prineville, OR. No, make that Redmond, OR. Facebook couldn't get good long-term weather data for Prineville, so they used weather data from Redmond instead (see page 8 of this PDF).

Prineville and Redmond are about 16 miles apart.

Does 16 miles make a big difference in the weather? 3 miles?

Boulder Creek, CA, averages 52 inches of rain per year. San Jose, CA, 16 miles away, averages 14 inches of rain per year. That's a big difference!

While designing our new server room, I searched for nearby long-term weather data. The nearest official weather station is less than 3 miles away. That station averaged about 40 hours per year above 90°F.

Last year I put a weather station on the roof. The roof is a bad place for measuring weather, but a good place for measuring the air intake to our new HVAC system.

On hot days, our weather station was consistently 10°F cooler than the official weather station. We're closer to the ocean, cooler and windier than the other side of town.

We could have 0 hours per year above 90°F, and no need for any cooling! If I had longer, on-site weather data to back that up, I could have saved a lot of money by not buying an air conditioning system.

If you are thinking about building a server room or data center somewhere, start with local, long time span weather data. But you should install your own high-quality weather station on-site, and start recording. Now.

What about climate change? On average, most places will get hotter, a few places will get cooler, but everywhere will get more extreme weather. Add whatever fudge factor makes you comfortable.

"Climate is what you expect, weather is what you get."
Robert A. Heinlein, The Notebooks of Lazarus Long.

Post-RAID storage designs

2011-03-21T12:31:00.000-07:00

Large RAID arrays are a pain. Maybe we're better off without them.

Here are some preliminary thoughts on a post-RAID storage design that meets our needs.

We've had big problems with RAID arrays, including multi-week rebuilds, and multi-week tape restores.

Now I'm designing a server refresh, and taking a fresh look at storage.

We need:

600 GB of ultra-fast storage
9 TB of fast storage
50 TB of slow, reliable storage.

My initial designs specified:

600 GB PCIe Flash card for ultra-fast storage.
600 GB, 15K RPM, SAS drives with RAID 0 for fast storage.
2 TB, 7.2K RPM, SAS drives with RAID 6 for slow storage.

Speed isn't everything, if your system is down for weeks recovering from RAID failures. We also want to recover quickly from disk failures, and from corrupted or accidentally deleted files.

So how about this design instead?

600 GB PCIe Flash card for ultra-fast storage.
600 GB, 15K RPM, SAS drives as JBOD for fast storage.
2 TB, 7.2K RPM, SAS drives as JBOD for slow storage.
More 2 TB, 7.2K RPM, SAS drives as JBOD mirrors of each of the above

Daily rsync mirror of the ultra-fast storage to a 600 GB mirror drive partition.
Daily rsync mirror of each fast disk to a 600 GB mirror drive partition.
Daily rsync mirror of each slow disk to a mirror drive.
Run the rsync jobs at about 6 pm, Monday through Friday only.

This design needs more slow disks than the first design, but we gain several from dropping RAID 6.

What happens if a disk fails?

We need to recognize the problem before the next rsync job runs
Halt that particular rsync job
Redirect to the mirror partition or drive, possibly with reduced performance
Schedule down time to power cycle the failed drive, and replace if needed
Copy the mirror partition or drive back to the original drive
Redirect back to the original drive
Restart the rsync job

Step 5 is where we save a lot of time compared to rebuilding or restoring a large failed RAID array. All the other steps are identical to RAID failure, or don't take much time (e.g. halting and restarting rsync jobs).

Even if we needed to recover one disk from a backup tape, that will take much less time than recovering a large RAID array from many backup tapes.

We would have similar steps for recovering a corrupted or deleted file. Ideally, we would have daily ZFS snapshots, but that has other issues.

Why the specific rsync days and times?

Virtually all of our work is done during normal working hours, 8 am to 6 pm Monday through Friday.
We typically recognize and begin restoration procedures only during normal working hours.
We don't want our mirrors to reflect corrupted disks or files before we get a chance to recognize and restore.

That means we don't want rsync jobs running on weekends, we don't want rsync jobs running after long periods of low file activity, and we really don't want RAID 1 which instantly mirrors problems to the other disk!

This design also reduces our risk from total RAID array recovery failure; i.e. everything goes wrong and we are unable to recover any data on a large RAID array. With this design, a double or triple disk failure only loses the data on those disks, rather than the entire array. And others have observe correlated or cascading disk failures due to identical designs, same manufacturing batch, and identical operating environments.

I'll have to think through all the implications of this design for a while.

Mac mini in a data center

2011-03-15T10:10:00.000-07:00

Apple dropped the Xserve, pushing the Mac Pro and Mac mini as server alternatives. We need to run many Mac-based applications in our data center, and we've chosen Mac minis.

How do you rack mount a Mac mini? How do you provide IEC C14 or 208V power to a Mac mini? How do you add an Ethernet management port to a Mac mini? How do you control a Mac mini through a KVM switch?

Old and New Mac mini important differences

Old Mac minis (original to mid-2010) are 6.5 x 6.5 x 2 inches with an external power supply.

New Mac minis (mid-2010 and newer) are 7.7 x 7.7 x 1.4 inches with an internal power supply.

Rack Mount Kits

State-of-the art data centers need to control airflow through cabinets and equipment. Strapping a Mac mini to a generic shelf doesn't cut it. Cooling air enters underneath, and exits through vents in the back.

MK1 Manufacturing makes a variety of rack mount kits, for both old and new Mac minis. We have several of these, for both old and new Mac minis. No problems.

Sonnet Technologies makes one kit, for the new Mac mini only.

Power Cords

Mac minis are sold in the USA with NEMA 1-15P or 5-15P power cords which support 120V power only. Some data centers require IEC C14 power cords, or 208V power, or both.

The old Mac mini needs an IEC C5 to C14 power cord.

The new Mac mini needs an IEC C7 to C14 power cord.

Mac minis draw up to 110 watts, so 2.5 amp cords should work.

Stayonline has the correct power cords in several lengths and gages, and adapters if you want to use standard power cords. We haven't tried these yet.

These power cords might be available from other sources, but I couldn't find them easily.

Ethernet Management Port

You can remote control a Mac mini over the network using iChat, Apple Remote Desktop, VNC, or other applications. Some data centers require a second Ethernet connection for remote monitoring and management. Mac mini comes with only one Ethernet port.

Apple's USB Ethernet Adapter works great on a Mac mini to add a second Ethernet port. The Adapter supports up to 100BaseT; Mac mini has Gigabit Ethernet built-in (except the original 2005 model), so choose which network port needs more bandwidth. You can find USB Ethernet adapters from other sources, but you might have trouble with Mac support.

KVM Switch

Sometimes you need access to the console. In many data centers, you do that through a KVM switch.

Newer KVM switches support USB keyboard and mouse ports. Mac minis use USB for Keyboard and Mouse, so that's easy. You might get confused using a PC keyboard on a Mac.

Most KVM switches support analog VGA ports for video. Mac minis don't have VGA ports.

Mac minis have DVI, Mini-DVI, or Mini DisplayPort for video, depending on the model.

Apple sells VGA adapters for these ports, and you can find adapters from other sources.

You might need help to figure out which Mac mini models you have, and which VGA adapters you need.

Let the IPv6 regret storm begin ...

2011-03-08T09:22:00.000-08:00

We knew the IPv6 transition was going to be tough for technical reasons.

Now we have the first example of IPv6 security problems that will be tough to fix.

Anti-spam software will have a much harder time blocking spam precisely because of IPv6 3.4 x 10^38 addresses.

I'll bet there are a few more IPv6 security problems waiting to be found.

Let's set arbitrary deadlines to switch to IPv6 before we find or fix these problems!

Oh wait, we've already done that.

Oracle has ruined Sun

2011-02-28T23:06:00.000-08:00

Paul Krill at InfoWorld recently wrote that Oracle has saved Sun.

From my perspective – Oracle has ruined Sun.

We've been Sun customers since before Sun existed. We ran 2.x BSD on a DEC PDP 11/34, and 4.x BSD on a DEC VAX 11/750 in the early 1980s. Both operating systems were written in part by Bill Joy (a Sun co-founder) while he was a grad student at UC Berkeley. We purchased two Sun 1/100 workstations, serial numbers 170 and 171, picking them up at Sun's first location in Mountain View. SunOS 1.0 was a direct port of 4.x BSD. We've used Sun servers for over 25 years, and we were pretty happy for most of that time.

But now we are migrating to Red Hat Linux. Why?

Our application vendors have abandoned Solaris. The biggest shift happened during the prolonged acquisition of Sun by Oracle, which created too much FUD.

Oracle has made it clear that their future customers will buy Sun hardware only to run Oracle applications. We have no use for Oracle applications. Obviously, ISVs have gotten the same message.

Oracle has bungled our support contracts. We are current on all contracts, but Oracle doesn't think so. We have had serious problems open for several months without resolution.

Before Oracle bungled our contracts, we couldn't get a useful answer to several serious problems. The old Sun would escalate problems until they got solved. When Oracle support reaches the end of their canned script, the case is closed whether our problem is solved or not, with no escalation and no appeals.

We are getting no love from Oracle.

I am quite sad at this development. Solaris is a much better designed and engineered operating system than Linux will ever be. Hold your flames, you won't change my opinion.

I've watched this cycle in the IT industry before. I spent many years as an application programmer on Multics, a well-designed, well-engineered, and secure system. We switched to Sun because Multics was a marketplace failure. For years afterward, my Sun root password was "unixsucks".

Current password rules won't let me use "linuxsucks". Too bad.

Would I be happier if Sun had gone bankrupt? No.

But the current situation doesn't make me any happier.

Offsite backup scenario TCOs

2011-02-07T19:10:00.000-08:00

Calculated three year Total Cost of Ownership (TCO) for several backup scenarios and our environment and requirements.

Short version:

Tape still rules
SOHO disks are competitive
Enterprise disks are at least 2 times more expensive
Amazon S3 cloud storage is 8-10 times more expensive
Cost isn't everything

I read this Clipper Group paper In Search of the Long-Term Archiving Solution – Tape Delivers Significant TCO Advantage over Disk, and was blown away by their analysis.

I didn't believe that tape could be so much cheaper than disk. I decided to analyze various scenarios for my own setup.

Your environment and requirements will be different.

Here are the 3 year TCO results, from simple and local, to complex and remote:

$47,000 – LTO-5 tape loader on each server

$45,000 – Servers share local LTO-5 tape library server using Gigabit Ethernet

$51,000 – Servers share local LTO-5 tape libraries using Fibre Channel

$45,000 – Remote LTO-5 tape server

$44,000 – Remote Drobo-based disk server

$120,000 – Remote enterprise disk server

$497,000 – Amazon S3 cloud storage

$341,000 – Amazon S3 Reduced Redundancy Storage

$339,000 – Amazon S3 RRS using Import/Export for backups

I used list prices from well-known vendors, and Amazon's online S3 calculators. We never pay list prices for servers, disks, and tapes, so this makes Amazon S3 look even worse.

The first five options are about the same in TCO. Choosing between them comes down to other factors.

Enterprise disks are notoriously overpriced compared to consumer disks. Sometimes you get what you pay for. Sometimes.

I am astounded at how many times more expensive the Amazon S3 options turned out. At Amazon's scale, they should be able to beat retail prices easily. I'm not seeing that in our environment.

Each of these scenarios has pros and cons which are not captured in TCO. I might discuss those in a future blog post.

Assumptions:

Offsite backups of five servers, 20 TB net available storage each.
Offsite backups ramp from 50 TB to 144 TB, in months 1 to 36.
About 4 TB per month incremental backups.
1 TB per month restores.
RPO about 30 days.
One year disaster recovery, i.e. save base plus one year of incremental backups.
No server rent or power costs. Those come out of Someone Else's Budget (SEB).
Free offsite server space and power – SEB.
No tape transport or storage costs – SEB.
Gigabit Ethernet WAN and Internet link.
No incremental WAN or Internet costs – SEB.
Replace everything after three years.
System administration costs roughly equal between options.
Backup software costs roughly equal between options.
Buy everything up front, including tape cartridges, except ...
Buy Amazon S3 services as needed.
No time value for money.

These assumptions roughly fit my environment and requirements.

Your assumptions will be different.

Long time no post ...

2011-01-19T22:59:00.000-08:00

Another multi-month break on blog posts. No dramatic medical issues this time.

My workplace is in multidimensional chaotic free-fall. In the past four months:

Major reorganization: nobody in middle management knows who reports to whom, and who makes key decisions.
Attempted takeover of departmental IT resources by headquarters
New accounting system with many bugs
New recruiting rules with many bugs
Abrupt reassignment of my boss – a Sunday night email effective Monday morning!
A series of temporary bosses, and no permanent replacement in sight.
One of my temporary bosses decided to make my life very miserable, in completely mystifying ways – and was rotated back to old job 5 days later.
Our major move is stuck in molasses. We are no closer to moving than we were 14 months ago when we signed the lease, and much grumbling about lawsuits.
The "movees" are really mad and barely talking to anyone else
One of the movees is in charge of requirements for our migration from Solaris to Linux, so that's going nowhere.
One key IT retirement in December
Attempted to hire a replacement, only to have him back out 10 days before he was scheduled to report to work.
Attempted to recruit another high-level IT position, only to get flooded by resume's from incompetent but "must consider" applicants because of our new recruiting rules.
That recruitment was handed off to our disgruntled movee mentioned above!
Reliable reports of HQ planning mass layoffs, in an organization that's had only three mass layoffs in 30 years.

My crystal ball for forecasting anything isn't just cloudy, it's shattered.

Hard to look forward when you are busy fighting for your career and trying to keep IT support from crashing around your ears.

Not looking for sympathy – except that you won't be seeing much new from me until some dust settles.

On a brighter note – Anathem is the best fiction book I've read in decades. Highly recommended.

Multi-level storage on the cheap

2010-11-16T17:55:00.000-08:00

Many vendors sell software or hardware to move your data automagically between different storage tiers. They charge a lot for this magic. Multiple storage tiers are almost synonymous with these expensive solutions.

We can get most of the benefits by using our brains instead, for far less money.

Virtually all of our server storage is files added and removed by real people, rather than databases automatically filling with new entries.

People are smart, and understand speed and limits – let users manage the storage tiers.

Imagine a server where users have several home directories:

/fastest/username
/fast/username
/slow/username
/slowest/username

Or:

/home/username/ [fastest storage tier]
/home/username/slow
/home/username/slower
/home/username/slowest

Slower tiers would have more capacity and larger disk quotas.

Users will naturally fill the fastest tier first, but as that fills, they will be forced to move their files to the slower tiers.

Each tier would be implemented with different speeds, costs, and technology.

Today, those tiers might look like:

Fastest – PCIe Flash drives
Fast – SAS MLC SSDs
Slow – 7200 RPM SATA RAID or ZFS filesystems
Slowest – Network storage to file servers

Other technologies could be used, like SSD-enhanced ZFS, Drobo storage products, or cloud storage. Each will have different cost-capacity-speed tradeoffs.

We must apply quotas at each tier to ensure some fairness.

We could expand storage at each tier as budget allows.

We could help users move stale files to slower tiers by sending periodic reports on the least-recently-used files at each tier.

We could use a similar strategy for web servers and other servers managed by sys admins. SAs can use their brains to optimize storage, especially if they have tools to understand I/O bottlenecks. But rules of thumb might be sufficient (and cheaper), e.g. put applications and swap on the fastest tier, put most-frequently-used data on the next tier, and put redundant copies on the slowest tier.

Rocky's Hardtack Recipe

2010-11-06T23:07:00.001-07:00

I really miss a bread-like food while backpacking, especially on longer trips. I tried many off-the-shelf bread, cracker, and tortilla variations, but they either tasted terrible or didn't last long in my pack or in storage.

Sometimes I would see references to hardtack, but hardtack is hard to find in local stores. I found a few recipes online, tried them, changed them, and came up with mine.

Don't let the name scare you, these aren't that hard (more like hard cookies), they taste pretty good, and they are easy to make for baking novices like me.

Rocky's Hardtack

1+1/2 cups milk
2 cups white flour
2 cups whole wheat flour
2 tablespoons honey
1+1/2 teaspoons salt

1. Mix the ingredients into a dough.

2 . Roll out to a thickness of 1/2 inch.

3. Cut into 40 mm x 90 mm bars.(*)

4. Place on a greased cookie sheet, and prick with a fork.

5. Bake at 375°F for 30-40 minutes, until golden brown.

Makes about 15 bars, about 2 oz net weight each.

If you intend to store them for several weeks:

Minimize contamination - use clean utensils, clean all work surfaces with soap and hot water, wash your hands well, and do not touch the bars at all after baking -- use clean utensils.

Allow the bars to cool to room temperature before packaging, to minimize trapped moisture.

Package the bars with a Seal-A-Meal or similar to exclude as much air as possible. Ziplocs are a poor second choice.

I normally carry two bars for lunch each day. They go really well with crunchy baked cheese.

Yum!

If you need real hardtack that can last for years, you can find many recipes online, including here.

You could also buy Sailor Boy pilot bread, available at Sam's Club, Winco, and Span Alaska Sales.

(*) Yes I know I've mixed Metric and US units. That's how I measured them. Most rulers have both units.

Lightweight backpacking in the early 1980s

2010-11-06T00:15:00.000-07:00

Some people write as if the lightweight backpacking movement of the past decade was never tried before, and that all backpacking before that focused on comfort at the expense of weight.

Wrong.

In the early 1980s, a wave of lightweight backpacking (LB) products swept through the industry for several years, then sank without a trace.

The early 1980s LB fad was a reaction to gear that had gradually acquired many marginally useless features (and more weight), or had been built bombproof for all the wrong reasons (with added weight).

Initially, I was skeptical, because I had finally acquired a bunch of expensive, heavy backpacking gear. Now the industry said "what we told you before was wrong, throw it all away and buy, buy, buy".

Piece by piece, I bought some LB gear, which served me well for many years. But by the 1990s when I needed to replace that worn-out gear, you couldn't find LB gear.

I'm not sure why LB gear vanished, probably for the same reasons it's a niche market today. Today, you can find lots of Internet-based small makers of LB gear, and lots of great LB advice in books and on the Internet.

Even today, it's pretty hard to walk out of REI with a base pack weight under 20 pounds, especially if you listen to the sales people. Even REI.com's "ultralight" section leaves much to be desired.

What was LB gear like in the early 1980s? Most of my LB gear has rotted or worn out, so this is mostly from memory. And many other lightweight products were available then; these are some of the products I owned or knew about first hand.

Let's look at the most important items -- pack, tent, sleeping bag, boots, and stoves.

Pack

Alpenlite made a sub-2 pound frameless pack I owned and used for several multi-day trips – the most comfortable pack I've ever used. The most sophisticated material in that pack was a sewn-in-place polyethylene stiffening sheet. Most of the pack was standard polyurethane (PU) coated nylon. That pack had thinly padded shoulder straps, a hip belt, a thinly padded back panel, two small front pockets, and a top pocket. What that pack didn't have was a lot of volume or useless features. With a smaller pack, I took less crap, my pack was lighter, and I was happier. I would still use that pack today if several parts hadn't rotted away.

Now I use an 18 oz Gossamer Gear Mariposa pack.

Tent

In 1979 I bought a 3.5 pound Moss Solus 2 tent with a lightweight PU coated rainfly and floor, a mosquito netting body, and one arched aluminum pole. That tent slept two grown men cozily. Unfortunately, the PU coated nylon fabrics used in those days would absorb moisture and break down into urea-related products. The coating would peel so the fabric would leak, and the tent would smell terrible. Eventually I threw it away.

If you wanted to go even lighter, Chouinard made the Pyramid and Megamid tarps, which weighed about 2.5 pounds but could shelter 2-4 people.

I just got a 29 oz Tarptent Moment.

Sleeping Bag

Western Mountaineering has made ultralight down sleeping bags since the late 1970s. I still have one from the early 1980s that weighs about 2 pounds and was good to about 30°F. Combined with the ubiquitous cheap blue foam, you could sleep comfortably for just over 2 pounds in late spring, summer, and early fall.

Now I use a 26 oz WM Megalite, with a 4 oz Gossamer Gear Nightlight pad that doubles as a pack frame.

Boots

The original Nike Lava Domes were basically running shoes with boot soles and reinforced toes and heels. My best friend worked in a backpacking shop and raved about them. In 1981, we met for a hike at Pinnacles National Monument, and he brought a pair my size. I literally took them out of the box and hiked 12 miles of rugged trails without a blister! Before that, my boots always required a painful break-in period, and still gave me blisters on long hikes.

Now I wear ASICS GT-2150 Trail running shoes.

Stoves

I never bought a lightweight stove in the 1980s. The closest I came was the Campingaz Bluet cartridge stove (very similar to this one), which was lighter and simpler than most stoves at the time. Decades later I learned that much lighter stoves like the Trangia and Esbit had been around since the 1930s.

Now I use a 3 oz Esbit pocket stove, but will try a home-made alcohol stove soon.

Now versus Then

Now you can travel even lighter than we could 30 years ago, using new materials like silnylon and inexpensive titanium, new designs, and new travel techniques. But the concept and market was definitely around in the late 1970s and early 1980s in the USA.

Which was probably an echo of an earlier lightweight backpacking revolution in the late 1950s and early 1960s.

Acknowledgements

Many thanks to Warren McLaren's outdoor gear history, Bruce B. Johnson's history of gear, and Monte Dodge's retro outdoor gear.

Thin clients on a separate network?

2010-10-27T11:03:00.000-07:00

Should we give everyone a thin client on a separate network for accessing sensitive applications?

We constantly struggle with new security requirements from headquarters. These requirements are one size fits all, meaning maximum security based on everyone using sensitive applications for personnel and payments.

And our scientists and technicians tend to use the same computers for research, downloading and installing new research-related software, and managing their project personnel and funds. This mix of sensitive and non-sensitive, need-to-minimize-risk versus need-to-take-more-risk activities on one computer is a security nightmare.

Since many hacks these days involve deeply penetrating computers and monitoring keystrokes, setting up locked-down virtual machines is hardly worth the effort.

So I'm considering setting up thin clients which can RDP into a Windows Server over a separate, locked-down network which has access only to the Server, and the Server has access only to the pre-defined sensitive applications. DoD, NSA, and other high-security outfits have been running separate computers on separate networks for decades.

What's different now is that the cost of thin clients, servers, and separate networks is approaching the lost productivity costs of locking down our research systems and arguing with headquarters.

This article from The Register triggered my thinking.

Time to define some requirements, create some preliminary system designs, and do some market research.

All of this might be moot if Headquarters doesn't allow us to apply different security standards to different sets of systems. Another Layer 8/9 problem.

Massive Disk Failure Deja Vu, Part 2

2010-10-26T17:21:00.000-07:00

So far, no further problems with this system.

Some corrections to the events reported in Massive Disk Failure Deja Vu, and followups on what we can or can't do to recover faster.

Corrected early event sequence:

One disk in one zpool went bad over the weekend, and started resilvering on the hot spare.
The other zpool complained about not having a hot spare.
The first zpool held the home directories of all users, and the resilver severely impacted performance -- logins wouldn't complete after 5 minutes!
Eventually, we reduced the performance impact, and allowed normal use while the resilver completed.

All of this is standard ZFS behavior, except for the surprisingly severe performance hit. The flurry of never-seen-before error messages, and many disgruntled users breathing down our necks, led to confusion about what actually happened.

The next day, when we discovered that a second drive in the same array went bad, we decided to be super-safe, and take the system down as reported earlier.

Followups on short term ideas to improve reliability:

Reduce zpool size – we decided this was too risky, and too much work, for the benefits we might get.

Aggressively scrub zpools -- we're currently scrubbing once each weekend, which takes 20+ hours with big performance impact. Not practical to scrub during working hours.

rsync from snapshot to another array -- we're already doing that to the extent we can, unless we add more storage. We'll adjust our rsync schedule to avoid some of the problems described here.

Backup to tape more often -- we're running full backups once per week that take 3.5 days to run, and can't run partial backups with our current software. Not practical to backup more often.

Bottom line -- we're not changing much.

We need to replace the storage on this system or replace the system, to get significantly better reliability.

Until then, we hope for the best.

A different angle on RTO, RPO, Backups and Restores

2010-10-25T00:13:00.000-07:00

When people design IT backup and restore processes, they typically focus on Recovery Point Objective (RPO), and Recovery Time Objective (RTO), with the goal of reducing both of those as much as the organization can afford.

Two hidden assumptions:

You instantly know when you have a problem.

You instantly initiate recovery.

Those assumptions are not always true!

What are the implications of getting these assumptions wrong?

Take this scenario as an example:

You have an RPO of 8 hours, and an RTO of 8 hours. So you carefully design a system that copies data off-site every 8 hours, using an expensive WAN link and duplicate storage, using something like rsync. In case of catastrophic failure, you can redirect to the off-site storage, while you rebuild your primary storage. You've done some testing, and believe both your RPO and RTO can be met.

Later, you suffer silent, massive data corruption that continues for 10 hours before you recognize the problem.

You are in big trouble. You've replicated the corrupted data off-site, and your RPO and RTO cannot be met.

Let's say your silent period is only 2 hours. You still have about a 20% chance that your RPO and RTO cannot be met, because off-site replication could have triggered during those two hours.

Let's say you instantly recognize your data corruption, but you can't initiate recovery for several hours. Same problems.

How could you not notice data corruption, or not initiate recovery, in time? Here are some examples:

Your system doesn't log enough information to warn of certain kinds of data corruption.

You aren't watching your system logs in real time for every possible type of data corruption.

You are watching your logs in real time, but don't recognize some particularly obscure error messages as data corruption.

You watch your logs 9 am to 5 pm Monday through Friday, but your off-site replication runs 24 hours per day.

You watch your logs 24 hours per day, but you can only initiate recovery 9 am to 5 pm Monday through Friday.

You watch your logs 24 hours per day, you can initiate recovery 24 hours per day -- but today several staffers are sick and nobody's available to watch logs or initiate recovery.

So what's the point?

We need to plan for delayed corruption recognition times, and recovery initiation times, as much as RPO and RTO, if you really want to meet your RPO and RTO objectives.

We need a couple more TLAs:

CRO -- Corruption Recognition Objective: The time period between the actual corruption or loss of data, and the recognition that recovery must be initiated.

RIO -- Recovery Initiation Objective: The time period between recognition that recovery must be initiated, and the actual initiation of the recovery process.

So, how can we improve CRO and RIO to minimize impacts on RPO and RTO?

Stay tuned for a future blog post.

Massive Disk Failure Deja Vu

2010-10-22T08:40:00.000-07:00

Last year, we suffered through extended downtime and data loss on our primary server due to multiple disk failures on a large ZFS RAID-Z2 array. The disk array vendor found no trouble in the array or drives, and my confidence in ZFS was badly shaken. If you need to refresh your memory: Part 1, Part 2, Part 3, Part 4.

It almost happened again a few weeks ago, with some bizzare new twists.

Two zpools share one hot spare drive. Our trouble started when that hot spare went offline over the weekend. Both zpools went offline, and we had an unusable system on Monday morning. Some frantic ZFS command-line efforts later, and we were back in operation, but resilvering both zpools. We crossed our fingers and hoped for the best overnight.

Next morning, another drive had gone offline. Too much like last year. We took the entire system down, reseated all the drives and array boards, and told ZFS to rebuild everything in sight. Like many systems, ZFS gives overly optimistic estimated completion times, but eventually the zpools are scrubbed. Based on last year's experience with file stumps (the file name is there but the file is empty), we use find and file to ensure every file still has reasonable contents. Then we rsync to a spare array, backup to tapes, and bring the system back online for users.

We exchange several rounds of cranky phone calls and emails with our array vendor again, including confusion over an array firmware update that was just released but turns out is irrelevant to our system. Our array vendor truly doesn't understand ZFS, and repeatedly pushes us to "upgrade" to their RAID 6 array to prevent further problems like this. We ignore them.

Bottom line:

No data lost.

About two days lost productivity for dozens of people.

No further problems after nine days.

Now what?

Something must change. We have experienced one catastrophe and one near catastrophe in less than one year on our most important server. Unfortunately, we have no idea what the root cause might be.

Since we can't discover the root cause, we must improve our ability to recover from similar failures in the future.

Some ideas I'm considering in the short term:

Reduce zpool sizes, to reduce rebuild times and reduce the data at risk from drive failures. This will reduce our net storage, but we're not full yet.

Aggressively scrub zpools to get early warnings of problems.

rsync to another array daily, so we can redirect to the second array and resume operation while fixing the first array.

Backup to tape more often. We backup using ZFS snapshots, so our backup window is not tightly constrained.

In the long run, we must consider replacing our slow, commodity-drive based system with a fast, enterprise-drive based system. Our data are mostly "cold", but we need faster backup and recovery times than we can get with our current setup. The purchase cost will be 2x to 5x higher, but we're losing nearly that much money from lost productivity and potentially from lost data, every year.

And we need to migrate from Solaris to Red Hat Linux in 2011. Oracle support for Sun hardware and Solaris software has been terrible. Third party software support for Solaris has already vanished for several of our key applications.

Of course, Linux means no ZFS, which means back to RAID 6 or some other scheme to improve the reliability of the data we store. Yes, FreeBSD has ZFS, but none of our third party application providers support FreeBSD.

I really hate to walk away from many ZFS features, including flexible zpools and partitions, snapshots, granular block verification, etc.

Moving off Solaris will end 25+ years of using Sun products for our group. We picked up two Sun-1/100 workstations with serial numbers around 170 from their roll-up door startup in Mountain View in the early 1980s, and have used mostly Sun servers ever since.

But our failure experiences, combined with terrible Oracle support and lack of third party software support, will force us off of Solaris+ZFS.

Love OpenDNS!

2010-10-21T10:47:00.000-07:00

I've been using free OpenDNS Basic at home for many months now.

We recently switched our Comcast visitors network at work to OpenDNS FamilyShield to enhance visitor security and reduce our liability for visitor misbehavior.

Our main corporate network has a Rube Goldberg DNS design so we can't touch that.

OpenDNS blocks lots of "bad" web sites, it's easy to setup (especially FamilyShield), and the Basic account gives me lots of flexibility.

At home I block many ad-serving domains to reduce the annoyance of ads, and reduce our vulnerability to ad-hosted malware.

admeld.com
apture.com
atdmt.com
constantcontact.com
dinclinx.com
doubleclick.net
googleadservices.com
imiclk.com
imrworldwide.com
interclick.com
pointroll.com
quantserve.com
questionmarket.com
revsci.com
revsci.net
scorecardresearch.com
tacoda.net
targetingmarketplace.com
viglink.com

I found these after watching OpenDNS logs for a while. YMMV.

Eventually, we'll further lock down our visitor's network by setting firewall rules that allow DNS access (port 53 UDP & TCP), only to OpenDNS FamilyShield servers - 208.67.222.123 and 208.67.220.123. That way you can't bypass OpenDNS by pointing to other DNS servers. Yes, there are other ways around this, but we're providing enhanced security with minimal effort.

Love OpenDNS. So does David Pogue.

Even More on new Data Center HVAC

2010-10-03T12:15:00.000-07:00

After another meeting with our HVAC subcontractor, I better understand the HVAC design for our new data center.

We will have two integrated systems; each system will have a blower added so we can move the CFM we require.

This design is even simpler than what I previously understood.

I will ask our HVAC subcontractor about using evaporative coolers instead of traditional AC compressors for those rare hot days. Should be even cheaper, hot days are almost always dry days, and we shouldn't need much water for those rare hot days.

We mounted a weather station on the roof to monitor our local conditions. Microclimates around here can vary significantly over a couple of miles, and the official weather station is about five miles away.

During the recent record-breaking heat wave, we were about 10 degrees cooler than the official weather station. Which means that my previous analysis of the days we'll need to cool outside air is very conservative.

We might need to cool outside air only a few hours of a few days each year.

IPv6 transition ordered

2010-10-03T12:04:00.000-07:00

We've been ordered to transition to dual-stack IPv6 plus IPv4 down to the client level in less than 4 years.

This will be a financial, operational, and security disaster.

For clear explanations of some of the major problems:

Iljitsch van Beijnum @ Ars Technica. Be sure to read the excellent comments.

D. J. Bernstein Old, but the major points are still valid.

Stuart Cheshire (9:23 QuickTime video) Co-author of many RFCs, and likes IPv6.

Carolyn Duffy Marsan @ NetworkWold. We must have NAT66 for address independence, even if it's ugly.

Besides those, some of our specific challenges include:

No funding. Once again, we're asked to make major infrastructure changes (like this), with no additional resources for training, equipment, software, services, or personnel. We're supposed to stop doing all that frivolous stuff we never had time or money to work on, and work on this in our spare time.

Need more bandwidth. IPv6 packets have more overhead, meaning we lose network bandwidth. Dual IPv6 plus IPv4 networks mean more routing and other protocol overhead. Some of our network connections run over Iridium at 2400 bits per second(!), so yes, this is a big deal.

Not looking forward to this.

Data Center Consolidation Gets Worse

2010-10-03T11:45:00.000-07:00

Our data center consolidation goals were finally revealed to us - four months after the decision was made.

Current: 350 "data centers" (3 or more servers)

Goal: 5 data centers

Impossible. Just impossible.

Data Center Consolidation?

2010-08-20T22:16:00.000-07:00

The Powers That Be have decided we have too many data centers, and we must consolidate. No doubt they will set a goal like 50% reduction, pulled out of thin air or from an airline's in-flight magazine.

Let's consider some of the hard problems with data center consolidation:

Nobody's repealed the speed of light. If we're forced to use a data center 1000 miles away (very likely), lots of applications will break due to latency. Add more latency from more hops through routers, firewalls, IDPs, and other security-boxes-of-the-week. I don't see any solution to this problem for certain mission-critical applications, even with magic WAN acceleration boxes.

Nobody's considered WAN upgrades and costs. If most of my servers move 1000 miles away, I'll need a massive WAN link upgrade to my local corporate node, and we'll need a massive corporate WAN backbone upgrade. We have constant battles over sharing the costs of our currently overloaded corporate WAN, I can just imagine the battles over the costs of a massive upgrade. And some locations can't get massive link upgrades. Most of our locations have one, maybe two WAN providers, usually just the local telco. Maybe they can't install a 1 Gbps WAN link to Timbuktu.

Nobody's considered the security implications. If you put all your eggs in a few baskets, you better guard those baskets really well. We usually don't, for many reasons. Yes, most of our small data centers have poor physical security. But most of them don't hold sensitive information, and we haven't had many problems with physical security.

We don't know how to run consolidated data centers. We have extremely distributed management, extremely distributed IT support, and extremely diverse IT requirements ("one of everything, please"). How will we manage and support all of that in a few locations, thousands of miles from the users and application experts? Cloud computing models are great – if you have relatively uniform IT requirements.

We don't know how to fund data center consolidation. We have extremely distributed IT funding. Consolidated data center funding will be, well, consolidated. Historically, unless we got new funding for a national IT project, we could not find a way to "pass the hat", or tax, or otherwise extract funds from possibly thousands of internal accounts.

We don't have funds for consolidation. We might save money in the long run, but nobody will front the transition costs. We expect 10% budget cuts for IT next year.

Nobody will do an apples-to-apples cost comparison. We never have, and I see no reason this time will be different. So we'll never know if consolidation saved money or not.

We have a bad definition of a "data center" - "three or more servers." I've had three physical servers running in my office sometimes! Do virtual servers count? Do desktops in a lab running automatically-enabled remote control, backup, file, print, and web servers count? How about telecom closets where each switch is an SNMP, SSH, and HTTPS server?

Many of these problems can be solved in many locations. Some of these problems are layers 1-7 (technical), others are layer 8 (financial), or layer 9 (political) (get the t-shirt!), and those are usually the hardest to solve.

I'm not opposed to data center consolidation, just the dumb way we're approaching it. Setting arbitrary goals to reduce the number of data centers without knowing how we'll solve all of these problems is stupid.

My suggestions:

We should go after the low hanging fruit. Our nearest regional headquarters has nine "data centers" (with three or more real servers) that I know about, and that regional hq is the smallest of several.

We should make consolidated data centers attractive. No matter how hard you try, a distributed organization will find a way around consolidation orders.

My ideal setup:

A fast WAN link to easy-to-use, secure, reliable, inexpensive, internal or outsourced cloud computing and storage. I'll gladly move as many applications as I can to that cloud.

A small green data center on site, to run those applications that can't run in the cloud.

The first data calls have gone out. Let the games begin!

More on our new Data Center HVAC

2010-08-20T20:28:00.000-07:00

I had more time to digest the HVAC proposal for our new data center and wanted to explain the new design.

Background

During design discussions with our mechanical sub-contractor, he was not sure our radical requirements could be met with off-the-shelf equipment. The initial design was overengineered using traditional requirements. After further discussions, he got excited about solving our problem, and came up with a better design.

Design

Two variable-speed blowers with bypass, will supply outside air 99% of the time, mixing data-center-warmed air when the temperature drops below 60°F, varying speeds as our CFM requirements change.

Two package air conditioners, will cool 1% of the time, when outside temperatures exceed 90°F. These could be downsized because air conditioners run much more efficiently at high temperatures

Simple. Not as simple as a custom or packaged solution that does both outside air and air-conditioning. But a custom solution would cost much more, and a packaged solution isn't available.

Why two units? Redundancy. Each unit is about half the required capacity. If one unit needs maintenance or fails, we still have half capacity until repairs are completed.

As I wrote this, got the idea to suggest we install just one larger blower and one larger air conditioner. We'll still have redundancy! Should be cheaper, simpler, and more efficient using larger fans. That's if larger units are available that will work with this design.

Note that HP designed their POD containerized data center to run at 90°F. This YouTube tour explains why this works, starting at 2:50. Our design also maintains strict separation between hot air and cold air using chimney cabinets, while using larger, more efficient fans on the roof.

100 mile Backpack Trip in Santa Cruz Mountains

2010-08-18T17:15:00.000-07:00

You can backpack for 10 days in a loop in the Santa Cruz Mountains, between Silicon Valley and the Pacific Ocean, hiking 8-12 miles per day. This trip is about 100 miles, depending on which trails you take, and which maps and signs you believe!

Trip Overview
Click to Enlarge
Image from Google Earth

After day hiking in these mountains for over 20 years, I realized that you could backpack a long loop. I haven't seen this loop described anywhere else. I backpacked this trip alone in 2002, 2003, and 2008.

This trip is mostly on well-maintained trails and dirt roads, with a moderate amount of climbing and dropping. You will hike about 4 miles alongside paved country roads with few cars.

Each night you camp in a campground, with an outhouse or flush toilets. Most camps have drinkable water. Jay Camp has pay showers nearby. Costanoa Resort has many luxuries, about halfway through the trip. Backcountry camping is not allowed. You must reserve most campsites by phone. Some camps are booked many weeks in advance.

By picking up supplies at Costanoa Resort, you carry only five days of food at most. You pass reliable water sources every day, though you need to carry water overnight three times. You need to carry a stove, open fires are mostly prohibited.

Cell phone coverage is very spotty in these mountains, but you pass working pay phones most days.

You will be hiking through mixed redwood forests most of the time, so you really appreciate the few scenic overlooks.

I've backpacked this trip in the spring and in the fall. You can go for days without seeing other hikers, yet you are surrounded by 7.4 million people in the greater San Francisco Bay area.

Spring and fall temperatures range from the 30°s F to 80°s F, with rain possible. Tent recommended for the rain and bugs. Mosquitoes are a nuisance at dusk and dawn. Ticks, scorpions, and other night time visitors can cause problems. I stored food in my tent at night without problems, but raccoons and mice could cause problems if you are careless.

Getting There

You can start at many different trailheads and hike either direction. The recommended trailheads and direction minimize food and water carries. Finding good water is surprisingly hard in these rainy mountains.
You could have someone drop you off and pick you up at the recommended trailheads, which have no overnight parking.

You could take SamTrans buses to within a half-mile of the Arastradero trailhead, and walk the rest of the way. You could take Santa Clara Valley VTA buses to within a couple of miles of the Arastradero and Rancho San Antonio trailheads, and walk the rest of the way. You could take Santa Cruz Metro buses to Big Basin headquarters or Waddell Creek, but you must carry a lot more food.

For other trailheads, you can request overnight parking nearby when you make camp reservations. Some parks get confused when you state that you are hiking through and don't need parking!

Maps

Enid W. Pearson Arastradero Preserve – City of Palo Alto

Foothills Park – City of Palo Alto

South Skyline Region – MROSD

Castle Rock State Park – State of California

Big Basin Redwoods State Park – State of California

Costanoa trails map – Costanoa Resort

Butano State Park – State of California

The Basin Trail Easement – Sempervirens Fund

Pescadero Creek County Park – San Mateo County

Portola Redwoods State Park – State of California

Rancho San Antonio County Park and Preserve – MROSD

Two maps give a good overview of the hike, but they have many frustrating errors:

Trail Map of the Santa Cruz Mountains, Map 1 – Sempervirens Fund

Trail Map of the Santa Cruz Mountains, Map 2 – Sempervirens Fund

Day 1 - Arastradero to Black Mountain

Start in Palo Alto at Pearson-Arastradero Preserve, and hike towards Foothills Park. Non-residents are allowed to hike through Foothills Park (1). Hike through into Los Trancos Open Space Preserve, then into Monte Bello OSP, and camp at Black Mountain Backpack Camp. Black Mountain has non-potable water which you should filter, treat, or boil.

Day 2 - Black Mountain to Castle Rock

Hike down Stevens Creek and up through Upper Stevens Creek County Park to Highway 35 - Skyline Blvd. Continue southeast, east of Skyline, through Saratoga Gap OSP, to the intersection of Highways 35 and 9 at Saratoga Gap. Cross Highway 9 and follow the Bay Area Ridge Trail east of Skyline to Castle Rock State Park. Camp at Castle Rock Trail Camp, with drinking water.

Day 3 - Castle Rock to Waterman Gap

Hike to the headwaters of the San Lorenzo River and the Skyline-to-the-Sea Trail. You will follow this trail to the sea for the next few days. Camp at Waterman Gap Trail Camp, with drinking water.

Day 4 - Waterman Gap to Big Basin

Follow the Skyline-to-the-Sea Trail to Big Basin Redwoods State Park. Park headquarters has a small camp store with fast food. Camp at Jay Trail Camp, with drinking water.

Day 5 - Big Basin to Pacific Ocean

Follow the Skyline-to-the-Sea Trail to the Pacific Ocean at Waddell Beach. After your ritual toe-dip, return the way you came. Load up on drinking water at the horse camp, and camp at Twin Redwoods Trail Camp or Alder Trail Camp. You could treat and drink Waddell Creek water, but I don't recommend it.

Day 6 - Pacific Ocean to Costanoa

Cross Waddell Creek, hike up West Ridge Trail, over Chalk Mountain, and down Whitehouse Creek, to Costanoa Resort. You can sleep in luxury, or pitch your tent. I like the Douglas Fir Cabins. Costanoa has a very small store and restaurant.

Once, my wife met me here with supplies for the second half of the trip. Other times, I dropped off supplies packed in two 5 gallon buckets with lids, which the resort stored until I arrived.

Caution: some maps show a trail between Whitehouse Creek and Old Womans Creek, crossing the inland portion of Ano Nuevo State Park. Don't take it! In 2003, that trail was badly overgrown, and crossing Old Womans Creek was quite difficult. In 2008, California State Parks had placed No Trespassing signs in that area.

Day 7 - Costanoa to Butano

Take the back roads through Costanoa, parallel to Highway 1, to the intersection of Highway 1 and Gazos Creek Road, where you might find a small convenience store and restaurant open. Gazos Creek is your last chance for a short side trip to the Pacific Ocean. Follow paved Gazos Creek Road and Cloverdale Road (you might see a dozen cars on a weekday) to Butano State Park. Load up on drinking water at the park entrance or one of the car camps along the way. Camp at the Trail Camp. An unreliable creek is about 1/2 mile away and a few hundred feet lower.

Day 8 - Butano to Slate Creek

Take Butano Fire Road through Cutter Scout Reservation to the end at China Grade Road. Take the Basin Trail Easement down into Pescadero Creek County Park. Cross Pescadero Creek into Portola Redwoods State Park. Load up on drinking water at park headquarters. Camp at Slate Creek Trail Camp. An unreliable creek is about 3/4 mile away and a few hundred feet lower.

Day 9 - Slate Creek to Black Mountain

Hike up to Long Ridge OSP. Follow the Bay Area Ridge Trail to Skyline Ridge OSP. Cross over Skyline Blvd into Montebello OSP. Camp at Black Mountain Backpack Camp again.

Day 10 - Black Mountain to Arastradero or Rancho San Antonio

You can return to your starting point at Pearson-Arastradero Preserve in Palo Alto, or hike over the top of Black Mountain to end your trip at Rancho San Antonio County Park near Cupertino.

Miles and Trails

Here are the miles and trails for my 2008 trip. Slate Creek camp was closed for the season, so I used Tarwater Camp in Pescadero Creek County Park instead, a few miles out of the way. Slate Creek is much nicer!

Distances shown on maps, signs, and my hiking-pace estimates differ, sometimes wildly. I've always found the MROSD maps to be quite accurate. Where I had any doubt, the miles shown are my best estimate.

Start – Pearson Arastradero Preserve

0.1 – Parking lot trail
0.5 – De Anza trail
1.4 – Arastradero Creek Trail
0.3 – One-way Road
0.3 – Road to Orchard Glen
0.4 – Wildhorse trail
0.8 – Costanoan Trail
1.4 – Los Trancos Trail
0.5 – Bay to Ridge Trail
2.1 – Page Mill Trail
0.3 – Canyon Trail
0.8 – Bella Vista Trail
0.7 – Old Ranch Trail
Night 1 – Black Mountain Backpack Camp
1.0 – Indian Creek Trail
2.3 – Canyon Trail
1.1 – Table Mountain Trail
2.7 – Un-named Trail
1.7 – Saratoga Gap Trail
Cross Highway 9
1.7 – Skyline Trail
Cross Highway 35
1.8 – Loughry Woods Trail
0.2 – Service Road
Night 2 – Castle Rock Trail Camp
0.6 – Frog Flat Trail
0.4 – Saratoga Gap Trail
2.1 – Travertine Springs Trail
0.8 – Saratoga Toll Road Trail
0.4 – Beekhuis Road Trail
2.6 – Skyline to the Sea Trail
Night 3 – Waterman Gap Trail Camp
4.3 – Skyline to the Sea Trail
Cross China Grade Road
4.3 – Skyline to the Sea Trail
0.4 – Sequoia Trail
Night 4 – Jay Trail Camp
0.4 – Sequoia Trail
0.9 – Skyline to the Sea Trail
Cross Middle Ridge Road
9.4 – Skyline to the Sea Trail
Pacific Ocean
2.0 – Skyline to the Sea Trail
Night 5 – Twin Redwoods Trail Camp
0.1 – Skyline to the Sea Trail
1.0 – Clark Connection Trail
4.2 – West Ridge Trail
1.4 – Chalk Mountain Road
1.2 – White House Ridge Trail
1.5 – White House Canyon Road
1.2 – Costanoa trails
Night 6 – Costanoa Resort
1.4 – Old Highway 1
2.1 – Gazos Creek Road
1.2 – Cloverdale Road
0.3 – Butano SP road
1.0 – Six Bridges Trail
0.4 – Campground Trail
0.3 – Goat Hill Trail
1.6 – Doe Ridge Trail
0.2 – Olmo Fire Road
0.9 – Indian Trail
0.5 – Trail Camp spur trail
Night 7 – Butano Trail Camp
4.7 – Butano Fire Road
1.8 – Basin Trail Easement
2.2 – Butano Ridge Trail
0.1 – Old Haul Road
0.9 – Bridge Trail
Night 8 – Tarwater Trail Camp
1.4 – Pomponio Trail
Portola State Park HQ
0.2 – Service Road
0.1 – Old Tree Trail
4.2 – Slate Creek Trail
2.1 – Ward Road
2.0 – Long Ridge Road
2.9 – Ridge Trail
Cross Highway 35
0.4 – White Oak Trail
0.9 – Canyon connector trail
0.2 – Canyon Trail
1.0 – Indian Creek Trail
Night 9 – Black Mountain Backpack Camp
0.2 – Indian Creek Trail
0.2 – Monte Bello Road
4.1 – Black Mountain Trail
3.0 – Chamise Trail
0.6 – Mora Trail
1.0 – Lower Meadow Trail
End – Rancho San Antonio CP
105.3 Miles Total

Many shorter and longer variations are possible. Grab a bunch of maps and plan your own trip soon!

(1) Palo Alto Municipal Code 22.04.150(b):

For the purpose of using the Bay-To-Foothills trails in the park, no person who is not a resident of the city shall enter or exit the park except on foot and at the designated entry and exit locations on the park boundaries shared with the Enid Pearson Arastradero Preserve and the Los Trancos Open Space Preserve.

Comcast update

2010-08-17T18:45:00.000-07:00

In my first Comcast post, I was surprised at the range and price/performance of Comcast's offerings in our location, and thought we might take advantage of some of them.

Time for an update:

Still very little demand, we don't have it.

WAN

After months of negotiations and internal roadblocks and rule changes, we got caught in an end-of-fiscal year freeze. No Comcast Gigabit Ethernet Private Line WAN until sometime in early 2011.

Too bad. We had pulled extra cabling through conduit for that connection, so it would survive construction. Stuck with only 45 Mbps until then.

Voice

Headquarters made us an offer we couldn't refuse, so we went with Avaya VOIP. If that doesn't work out (another site has had multiple multi-hour outages), we will re-consider Comcast phone lines.

Internet

We dumped DSL, pulled coax from the Comcast MPOE to our telecom closet, and installed Comcast Business Class Internet. Our visitors network has been faster and more reliable for several months now. I'm happy :-)

Staying warm on the river

2010-08-17T18:28:00.000-07:00

On a guides mailing list we were asked for tips on keeping paddler guests warm late in the day.

Here are some of mine:

Preventing hypothermia is far, far easier than treating hypothermia. Anticipate shady-cool, windy, wet conditions late in the day, particularly as the days get shorter. Keep your guests warm, dry, well-hydrated and well-fed throughout the day.
Check your guests warmth, don't rely on your own comfort level. You might be built like a walrus, acclimated, warm and toasty, while some of your guests are turning blue. Ask each guest (not the "group") if they are warm enough -- some will lie and say they are "fine". Look for wet cotton, goose bumps, shivering, blue skin, lethargy, slurred speech.
Keep a special watch on kids. They resist wearing warm clothes, and can go from bundles of energy, to blue and shivering in the blink of an eye when their reserves run out.
Hang a keyring thermometer from your PFD for an objective measure of temperature. Reminds me to check with my guests.
Put your guests in garbage bags. Bags block the wind and keep the heat in, while taking very little space. Tear holes for the arms and head, slip over your head. Worn over or under PFDs, endless discussions on the pros and cons.
Wet cotton must be removed, even under splash jackets, wetsuits, and garbage bags. Guests will be warmer with bare skin than wet cotton, especially when you have upstream winds.
Work your guests harder, they will warm up, and usually have more fun. Call more crew turns. Catch and release some eddies. Find a wave to surf. Pretend to be a salmon swimming upstream. Spin the boat in circles. Use your imagination, make it fun.
Keep your boat in the sun as you float. Work your crew to catch those last slivers of sunlight.
Keep your guests dry - stop the water fights, avoid the big splashes, keep guests in the boat and out of the river. Sneak around the big holes and waves instead of going for the wild ride.
Tell your guests: "If you're gonna FALL, fall INTO the BOAT" to keep them drier.
Move the cold, wet, shivering bow paddlers to the middle of the boat, and let warmer guests have some fun up front.
Encourage your guests to drink water. Not intuitive, but well-hydrated people stay warmer, even drinking cold water. Never river water or alcohol.
Feed your guests. Provides both a calorie boost and a psychological boost.
Keep your guests engaged by singing songs, playing games, etc. Don't let them hunker down quietly shivering.
If someone is shivering, deal with it now. Don't wait for things to get worse. Stop the trip if you must.
At takeout, get shivering people out of wet clothing and dried off ASAP. Then either work them hard loading gear, or get them into a car, don't let them stand around. Put someone in charge of dealing with cold wet guests, and let the rest of the guides deal with logistics. Send one car ahead with the coldest guests.
At takeout, keep PFDs on until the last minute for extra insulation and wind blockage.
At takeout, have snacks and drinks ready, usually in the shuttle vehicles.
Head guides must check in with all guides, and take a look at all guests, to make sure all of these things are happening.

Fiber to the desktop? Not anytime soon

2010-07-08T16:46:00.000-07:00

For decades, some vendors have pushed fiber to the desktop, instead of copper.

The advertised advantages of fiber have been:

Fiber is "future proof" – you don't need to rip and replace every 10 years
Fiber has higher bandwidth
Fiber is immune to RFI
Fiber is almost impossible to tap
Fiber can run longer distances easily

In reality:

Fiber has been through many "standards" in two decades. Multimode, singlemode; 8.3, 10, 50, 62.5, 100 um "outside" diameters; ST, SC, FC, SMA, FDDI, and MT-RJ connectors; SX, LX/SM, LX/MM, LX10, ZX, and BX10 PHY standards for 1 gigabit; SR, LR, LRM, ER, ZR, and LX4 PHY standards for 10 gigabit; and more. Show me 10 year old fiber that can run 10 gigabit ethernet on easily available equipment without adapters!
Fiber has higher bandwidth at 100 meters – at much higher cost. And copper catches up in a few years.
Cat 5 and newer cable have virtually no RFI problems if installed according to specifications.
Tapping fiber or copper cables is the least of my security worries.
Yes, fiber can run longer distances than copper for the same or faster bandwidth. If you need distance, fiber is the only way to go.

And ...

Fiber is more expensive to install than copper, both for the cabling and the labor. However, the installed cost of Cat 6A is almost even.

Fiber is more expensive to use than copper. Since most end-user devices have only copper network interfaces, you need a fiber-to-copper converter or HBA for every device. And fiber network switches are many times more expensive than copper.

Most of these factors have not changed for decades, and won't change anytime soon.

We might pull fiber into a few offices for users with high-bandwidth needs.

But in general – Fiber to the desktop – bah, humbug.

Design choices for new space

2010-07-04T21:25:00.000-07:00

Made several important design choices for our new space over the past few weeks. Some of them surprised me.

Phones -- Avaya VOIP
With generous subsidies from HQ, we chose an Avaya VOIP phone system. I also negotiated a sweetheart deal where our enterprise telecom group is responsible for operations and security. The annual operating costs will be about 1/3 of an AT&T Centrex system.

We realized during this process that land-line telephones just aren't that important anymore. A friend of mine who works in R&D for one of the largest VOIP vendors told me he hasn't used his phone at work for months!

Telecom cabling -- Cat 6
Our cabling subcontractor convinced me to spec Cat 6 instead of Cat 6A cabling, because:

Cat 6A is much more expensive for both the cable and installation labor.
Cat 6A cabling is much larger in diameter, which increases costs in other areas, like conduits, raceways, and cable management; and creates problems trying to push Cat 6A into our existing Cat 6 spaces.
If we need more bandwidth to specific offices or labs in the future, we can install fiber for about the same cost as Cat 6A. Fiber is not practical for widespread use until computers and phones ship with fiber network interfaces.

Data Center HVAC
Our HVAC subcontractor found some low-cost solutions to meet our requirements. Our design temperature is 85 F, but we will likely run at 90 F or 95 F, using outside air for cooling 99% of the year. This design dramatically lowered our backup generator load compared to conventional HVAC, so ...

Backup Generator -- 250 KW
We're getting a 250 KW diesel generator to cover the data center and telecom room IT loads and HVAC loads, in addition to critical lab loads and lab exhaust hoods.

We wanted a natural gas fueled generator for infinite fueling, lower CO2 emissions, and lower air pollution. Unfortunately, our local air pollution control district won't approve natural gas generators (go figure).

We discussed Bloom Boxes but they are too cutting edge for our landlord -- even though the supermarket down the street has several of them! The Bloom Boxes would be primary power, with local utility power as back power -- a reversal from the usual design.

Small green data center designs

2010-07-04T21:16:00.000-07:00

I had a very long, hard road designing my new, small data center. All the visible activity is in large data centers.

Ironically, if cloud computing really takes off, we could see lots of small data centers, with fat pipes to clouds.

Some tasks won't move to the cloud, ever. We'll need small data centers for those tasks.

Would be nice if someone created guidelines for designing small, green, data centers.

If I had a few months uninterrupted, I would write those guidelines, maybe as a Wiki so everyone could add knowledge.

Long Time Gone

2010-07-04T21:10:00.000-07:00

I know, everyone has lame excuses for not posting blog updates for months.

In my case, I really did get really sick and almost die -- twice.

Sorry, no sordid details. I'd like to maintain some privacy.

Massive Disk Failure, Part 4

2010-02-25T06:10:00.000-08:00

Our disk array vendor exercised and tested both the old array and the failed drives for three weeks.

No trouble found.

From what I've read, this outcome is not unusual. Way back in December, we power-cycled the array, and removed-reinserted the drives trying to clear these problems.

Remember the problem with our oldest Thumper? ZFS rebuild proceeded without drama, we swapped the bad drive for a spare, and life is back to normal. Maybe we got lucky in that case, and did not experience correlated drive failures. Or maybe luck had nothing to do with it, and Sun qualifies their drives better than our other array vendor.

We are actively pursuing a higher speed WAN line, like the offer we got from Comcast for 1 Gbps. If we can get a link that fast, mirroring between sites becomes thinkable for our ~200 TB pile of storage. Then, if the primary system fails, we can continue operating (with lower performance), until we rebuild.

I feel boxed in by the poor choices available from the IT industry:

- Pay through the nose for "enterprise" class storage (like 10x to 20x other solutions), but not sure we get 10x to 20x better reliability and service.

- Suffer potentially catastrophic disk failures and extended recovery times, using commodity storage, because drive manufacturers have pursued capacity increases without corresponding reliability and I/O speed increases.

- Pay through the nose for WAN bandwidth, unless we happen to have competitive choices for WAN providers. I don't get to choose our location, so relocating to a major metropolitan area with fiber sprouting from every manhole is not an option.

We can't pay through the nose for much of anything. My IT budget has been flat or declining since I took this job, but the scientists we support have gone from generating 100 MB per day in the mid-1990s, to 10 TB per day now. Add on weekly unfunded mandates from headquarters, and there are years where I have no discretionary budget.

So what's the solution for large, reliable storage on a shoestring?

I know, I know - pick any two (or one): cheap, big, or reliable. The people I support don't understand those tradeoffs. They are content to put all their data on $90 1 TB external hard drives they picked up at Best Buy on their way into work, and don't understand why I need to spend 10x that to get reliable storage.

Sometimes I don't understand, either.

Simple Change Management system pays big dividends

2010-01-16T22:11:00.000-08:00

A few years ago, we implemented a simple change management system within our IT support group. We had seen problems with mismanaged changes, but we didn't need some elaborate expensive scheme complete with a Change Control Board as advocated by Big IT schemes.

How does it work?

A system admin fills out a simple Web form with a Change Request
The CR is emailed to all the other IT support people, and to me (the manager)
I approve or disapprove the CR in return email
Our webmaster adds approved CRs to our Change Schedule web site, which acts as both a forward look at upcoming changes, and a record of what we have changed.

What makes this scheme work is the rules that surround it:

All major changes to multi-user systems must have approved CRs before sys admins can make non-emergency changes.
Changes made without approved CRs must be undone immediately.
CRs must be sent about one week before the proposed change.
We have blackout weeks immediately before and during business-related crunch times. These total about 4 weeks per year.
Changes must be made on Tuesdays unless you have a really good reason for another day.
If you are making the change, you can't leave for vacation or other planned absence for a few days afterward.
The form asks for detailed information. If I don't get all that information, I'll deny the CR.

Why Tuesdays?

Mondays are bad because people need to get some work done after the weekend, and Mondays are often holidays.

Thursdays and Fridays are bad because often we see problems 1-3 days after a change is made (e.g. server crash), and we want sys admins and vendor support staff available. Most of our support contracts cover 8 am to 5 pm Monday-Friday.

That leaves Tuesdays or Wednesdays. We chose Tuesdays.

What's a Major Change?

A major change is any change that could affect two or more people. Over time, we refined what is a "major change" and what is "business as usual".

Major change examples:

Any server operating system or application update or install
Any firewall change
Any network change
Any patches or software that we push out to users, or advise users to install.

What fields are in the form?

Change title, including system name
Requester's name and email address
Names or descriptions of the systems affected
Planned date/time of change start
Planned date/time of change end
Short description of change for users
Long description of change for system admins
High/Medium/Low rating for User Impact
Backout plan, i.e. what will you do if the change doesn't work right?
Backout time limit, i.e. when will you "pull the plug" on the change and go back to what worked?
Benefits of change
Risks of change (I've rejected many CRs for glossing over this part)
Communication plan: Dates & times to send warning emails to users
Communication plan: Status update email after success/failure with brief explanation.

Believe it or not, our simple Change Request system is significantly better than virtually all of our Enterprise IT systems. I've seen no communication before major changes; incomplete descriptions of the changes with major impacts; changes dragged on for hours or days as they struggle to get to the goal rather than pull the plug and reschedule; and changes made without approval by anyone besides the responsible sys admin.

When we first implemented our CR system, our sys admins complained bitterly. Now, everyone uses it and expects everyone else to use it. A couple of times someone has forgotten, and that has caused a lot of problems.

As we've gained more experience with the system and trust in the sys admins, I've permitted some flexibility in notices, change days, etc., after getting good justifications.

How would I improve our system?

Automate the approval and posting of approved change requests. For example, when the CR email goes out, I should go to another web form, authenticate, and approve/disapprove/comment on the CR. If the CR is approved, the CR should automatically be added to the Change Schedule.
Automate the sending of warning emails to users from inside the CR system. The sys admin would compose warning emails inside the CR, and select dates/times to send the warnings. The CR system would automatically send the warnings. Sometimes sys admins forget to send warnings, or they are out sick.
Consolidate change warnings to users. Some Tuesdays we might have 6 pending changes. Consolidate warnings of those changes into no more than one email per day on the days before the change.

These changes would require significant programming to extend our current dirt simple system. We will explore using the Change Management tools built-in to our WebHelpDesk system instead. BTW, we really like WebHelpDesk. More on that later.

Simple Change Management works for us. Can't imagine not using a Change Management system now.

Focus on the big picture

2010-01-13T11:23:00.000-08:00

One of the new fallacies of the Googleized Intertubes is that anyone can become an expert on a topic with enough searching and reading web pages and listening to podcasts and watching videos.

I fall into this trap sometimes, especially when I'm involved in building design. In my experience, the architects, engineers, and contractors whom we depend on often recommend designs, systems, or products with which they are familiar, and are not always aware of alternatives.

So I throw myself into a Google/Internet frenzy, trying to find out more about the options. This leads to a false sense of expertise, and a tendency to say "I want design/system/product X" despite my lack of expertise.

I think a better approach is:

State our big-picture desires in the design documents (e.g. we need Gigabit Ethernet to the desktop now, 10 Gigabit in five years)
Use Google and the Internet to learn more about the options
Describe some of the options in the design documents
Require bids on some of the options so we can make cost/benefit choices
Challenge the architects, engineers, and contractors to come up with innovative designs and choices
Require the architects, engineers, and contractors to defend their choices, including costs and benefits

I'm struggling with design specification decisions for:

Network cabling to the desktop
Data center HVAC design
Data center network design
Data center power reliability (generators and batteries and flywheels, oh my!)

In part, I'm writing this blog post to remind myself to focus on the big picture.

We'll see if it works.

"Call and response" on the river

2010-01-09T17:10:00.000-08:00

I've taught hundreds of boats full of whitewater paddlers how to paddle. The basic paddle commands for Class III whitewater are:

Stop! - Stop paddling, take paddles out of water
Forward! - Everyone paddles the boat forward
Backpaddle! - Everyone paddles the boat backward
Left Turn! - Left side of boat backpaddles, right side forward paddles, boat turns left.
Right Turn! – Like left turn only reversed
Highside! - Everyone climbs to the high side of the boat to prevent wrapping around a rock.

Most crews, most of the time, are easy to train and pretty responsive. The training technique I use:

Get everyone to look at me
Explain one command
Demonstrate that command
Practice that command
Repeat all these steps for each command
After going through all the commands, run the crew through several commands in random order.

The commands that cause the most trouble are Left Turn and Right Turn. People have trouble remembering which side they are on, and what they should do (forward or backpaddle).

A few ways I reinforce Left Turn/Right Turn training:

Between Demonstrate and Practice, have everyone on that side of the boat raise one hand.
If they screw up a turn during practice, stop, point out the problem, and try again.
Early in the day, use an extended command, e.g. "Left Turn! (short pause) Left Side Back", until they get it.
After people move around in the boat, call a few turns to make sure everyone gets it again.

Pretty standard stuff, and this is what I teach new guides, and teach trainers.

These training techniques engage paddlers through listening, watching, and doing, on multiple occasions. This broad based approach is important, because people have different dominant learning styles for physical skills, and need repetition to hone those skills.

With some guides, some crews, and some really loud rapids, the paddlers in the front of the boat have a hard time hearing the guide shout commands. What looks like paddler stupidity or rebellion is often inability to hear or understand.

Yesterday, I thought of adding one more twist to this scheme.

Have the crew repeat the command back in slightly different form, a "call and response":

Guide says: Stop!
Paddlers say: All Stop!

Guide: Forward!
Paddlers: All Forward!

Guide: Backpaddle!

Paddlers: All Back!

Guide: Right Turn!

Paddlers: Right Side Back!

Guide: Left Turn!
Paddlers: Left Side Back!

Guide: Highside!
Paddlers: Everyone on the high side!

Adding this call and response could help paddlers:

Hear the commands
Do the correct command
Stay verbally engaged in the commands

Engagement is important. Experienced guides on rivers they know well can get down most of the river without their crews. But paddlers have more fun if they feel their efforts are important to getting down the river safely.

One more variation. Some rivers have "Quiet Zones" while rafting through suburban back yards. We teach our paddlers to use their "inside voice" in the Quiet Zones.

I've never seen this used on another boat. Maybe there is a good reason?

What do you think?

Massive Disk Failure, Part 3

2010-01-09T10:31:00.000-08:00

We've had a pretty good relationship over 10 years with the disk drive chassis vendor at the heart of this problem. We thought we had our bases covered with their extended warranty promising "overnight advanced replacement" of failed components.

Not quite. Try 2+ weeks to get the chassis replaced.

The problems cited by the vendor:

All of this happened in late December, and many of their staff were on "vacation" (or furloughed, we're not quite sure).
They did not have our particular chassis sitting on the shelf ready to ship. They had to build one.
After building our chassis, they had to "qualify" the chassis to make sure it worked right.
The chassis was ready to ship New Year's Eve. Oops. Chassis actually shipped on the Monday after.

Not much we can do about this. Our vendor let us down. Realistically, all we can do is not buy from them in the future.

Massive Disk Failure, Part 2

2010-01-09T10:13:00.000-08:00

Most of the dust has settled, though we are still waiting for root cause analysis from our drive/chassis vendor. They found nothing wrong with the first two drives that failed, but apparently this is a common finding.

Recovering from backup tapes was far more painful than anticipated. Despite weekly backups to tape, several people lost 2-3 weeks of work for various reasons:

Tapes written during our ill-advised "let's keep running during rebuild" phase had lots of mangled files and filenames.
The tapes before those just happened to be written during the first round of disk failures. More mangled files and filenames.
Discovering which files and names were mangled took lots of extra time and effort, including re-reading tapes.

In several ways we had a near worst-case-scenario for recovery time:

2 drives failed, rebuild took most of a week
2 more drives failed near the end of that week
Restoring from tapes took most of another week plus
"pax" tapes must be read from beginning to end while searching for mangled files.

For our 8 drive ZFS pool, our real-life bad-case recovery time is 2+ weeks.

We've had some heated internal discussions about what this implies for our three Thumpers (Sun X4500 servers). Those are currently configured with one giant pool of 40, 500 GB drives.

By my math, this means the real-life recovery time might be:

2 drives fail, rebuild takes 5 weeks
Near the end of the five weeks, one or more additional drives fail
Restoring from tapes takes another 5+ weeks

For a 40 drive ZFS RAIDZ2 pool, about 10+ weeks bad case recovery time, during which the system must be offline. If we had 2 TB drives, that could be 40+ weeks!

Even in our relaxed environment, that's pretty bad. But that 10+ weeks must be weighed against our anticipated failure rate for this scenario. We have about 11 operating years experience with ZFS across 4 servers. One disaster every 44 server years? With 4 servers, one disaster every 2.5 years?

Maybe worse than that. Our oldest Thumper just started throwing disk error messages. One disk only at this time, so we are at the wait and see stage.

My confidence in ZFS is somewhat shaken, but maybe the problem lies in our expectations.

Our recovery from a true physical disaster (fire, earthquake, flood, etc.) would take much longer than 10 weeks, since we don't have spare servers and data centers in place.

The obvious solution - servers mirrored and geographically separated - is not in my budget for the foreseeable future. And I'm not quite sure how you keep from mirroring garbage when the primary system goes flaky but not down.

We've got some thinking to do.

Massive disk failure

2009-12-29T09:22:00.000-08:00

I have been a big fan of Sun's ZFS file system for many reasons. We have about 100 TB of active ZFS storage, mostly ZFS RAIDZ2, which can correct from double drive failures.

We recently suffered the biggest storage failure we've had in decades. The story is still unfolding, but the beginning and middle have already been written.

Two drives in an eight drive pool failed over one weekend. This pool is on our most heavily used production server, so we decided to let the system rebuild onto new drives, while staying in production. The rebuild was going to take more than one week. During rebuild, many critical files were found to be corrupted. Despite daily ZFS snapshots, we were forced to restore these critical files from tape or other sources during the rebuild.

Before the week-long rebuild completed, two more drives failed. Now we were truly hosed. Four drives failing on one system in one week! We disabled multi-user access to the system. We began the laborious and lengthy process of restoring from tape to a previously unused, identical array. The restore should be finished today -- about 10 days later! The drive array vendor is taking this incident very seriously. The vendor has overnighted replacement drives, and a new chassis with power supply, so we can move our existing drives. We've seen bad power supplies take out multiple disk drives or other components before. The vendor plans to analyze our failed drives, old chassis, and power supply, to discover the root cause.

Here are some of our preliminary lessons learned from this incident:

With ZFS RAIDZ2, double drive failure means we must take the pool offline for rebuild. Do not try to keep the pool in production during the rebuild.
Smaller ZFS pools rebuild or restore from tape much faster than larger pools.
We must balance larger ZFS pools versus rebuild/restore times. Larger pools have lower parity overhead, but much longer rebuild/restore times.
Tapes are way slow. We need to find a better scheme for disaster recovery.

Possible alternative disaster recovery schemes include:

Restore from multiple tape drives simultaneously. We need a more complex backup/restore scheme, which might have other problems. And we are ultimately limited by the write speeds to disk, which might be exceeded by two or three tape drives.
Offsite backup/restore over very high-speed WAN connections to disk pools. Many variations on this theme, all involve very expensive WAN links, duplication of storage, and offsite data center space.
Offsite server mirroring. Duplicate your primary server, mirror contents in near-real-time, and fail over to the off-site mirror when your primary fails. Expensive duplication of servers, might need WAN speed increase depending on the bandwidth needed by the mirroring process or user access, and need offsite data center space.
Outsourced offsite backup/restore over very high-speed WAN connections. Like Amazon AWS. We must encrypt anything that goes offsite, and that might slow down the process too much to be practical. And lots of money needed for the offsite storage and the WAN link.
Outsourced offsite server mirroring is not an option, due to our security requirements.

We could accept the risk of extended downtime every decade or so, and continue our existing practices, with some operational modifications.

We'll have some strategic discussions after the dust settles to decide how to proceed.

Comcast?

2009-12-22T23:41:00.000-08:00

Heard from our building manager that Comcast had pulled cable into our building. So I called Comcast to find out what they might offer for our existing needs, and for building our our new space.

I was surprised.

We don't have much need for Cable TV, but apparently we can get basic analog cable for a small additional charge if we get WAN, voice, or Internet.

WAN

We have an AT&T DS3 connecting us to our regional hub and on to the corporate network, and through that to multiple Internet POPs. We pay about $3,000 per month for 45 Mbps. We have two AT&T T-1 lines (1.5 Mbps) as backups. In six years, we've had only one extended outage on the DS3 outside of our control, and the same fiber cut took out our T-1s too. Sigh. At least we dumped our Cisco routers, and use much cheaper and simpler LAN extenders plus switches running Spanning Tree Protocol to cut over to the T-1s when the DS3 does fail by itself (e.g. we pull the wrong plug in a telecom room).

Talking to one part of Comcast, we can get 100 Mbps point-to-point Ethernet for $3,000 per month, and 1 Gbps for $4,000 per month. Wow.

Unfortunately, talking to other parts of Comcast, they claim we have no Comcast fiber in this county, which means nothing faster than what we have. I'm still working with Comcast rep #1 to see how far we can push this.

Yes, we could use a speed boost - we have several users moving multi-terabyte data sets around. That's when I start mentioning "a station wagon full of tapes hurtling down the highway", and start wondering about van-mounted file servers with generators and long Ethernet cables.

Voice

For 240 phone lines, Comcast will charge us about 2/3 of AT&T, plus more features and unlimited domestic long distance. We don't need most of the features. The long distance isn't a major concern, our corporate contract costs about 2c per minute, and the bills don't come out of my budget. But the monthly service charges certainly do. Downside is we'll need 30 Comcast 1U VOIP boxes in our telecom room (8 lines per box). Each box has a battery for 16 hours of service.

I've mentioned this to friends, who promptly point me to multiple online horror stories about residential Comcast Digital Voice. Comcast makes big distinctions between their consumer and business class services, often with higher prices for business. I should get a few voice lines and see how it goes before jumping to 240 lines.

Internet

We have DSL Internet for visitors, to keep them off our corporate network. DSL can work for years with no problems.

But when something goes wrong, getting DSL fixed can be almost impossible. We're on DSL provider #2, with the same old issues. The line is owned by AT&T, wholesaled by an unidentified third party (though traceroutes give me a pretty good guess), and retailed by a local ISP.

On another DSL from the same ISP, we lost connectivity for several days, ripped apart our inside wiring on a wild goose chase, bought and swapped four DSL modems, and waited overtime for one AT&T truck roll. Three weeks later we had 2/3 of our old bandwidth. Last proposal from the ISP was to keep swapping DSL modems and scheduling AT&T truck rolls, with circular finger pointing from everyone involved. I said enough is enough!

So we will switch to Comcast Business Class High Speed Internet to replace both DSLs. Can't be much worse, and at least the finger pointing should be minimized when something goes wrong.

Tapes are not for long term storage

2009-12-22T23:07:00.000-08:00

Tapes are much more energy efficient than even the best MAID disk storage system.

However, until you've had to recover data from tens of thousands of rotting tapes which need tape drives that are no longer manufactured -- don't talk to me about energy efficiency. We need to keep lots of digital data "forever", but we rarely get enough money to do that well.

I can tell many strange stories about tape rot and recovery, but one story is especially absurd.

We store 9-track tapes hanging by hooks from specially designed racks. Tapes have sticky labels so we can identify the contents. After a couple of decades, the glue no longer holds the labels onto the tapes. Every morning, we find piles of these labels on the floor, and unlabeled tapes on the racks. Most of these unlabeled tapes go straight to the recycle bin, because without the labels, the tapes are worthless.

Tape capacity is not keeping up with disk capacity. 2 TB 3.5 inch hard drives are smaller and almost as cheap as 800 GB (uncompressed) tapes. Moving data on and off tapes is much slower than disks.

BTW -- tape compression is almost useless. All of our space-hogging data, like images, video, and many scientific formats, are already compressed, and can't be recompressed, so we never see the 2:1 or even 2.5:1 compression ratios claimed by various tape vendors.

My disk versus tape philosophy goes something like this:

Tapes are easy to put away and ignore
Recovering data from old tapes is expensive, and easy to postpone ("next year")
Ignore old tapes long enough, and you lose the data

Big piles of spinning disks are hard to ignore.
Eventually disks fail and can't be ignored.
When disks fail, you replace them, move the data, and your data are safe for another few years.

Therefor:

Long term data storage is always on disks, because you are forced to deal with the data.
Disaster recovery storage can be on tapes, because after a year or so, the value for disaster recovery is almost zero.
Tapes have value for disaster recovery, data transfer, and other uses, so we will use tapes for the foreseeable future.

However:

Using disk drives or remote storage for disaster recovery could eliminate tapes. Just waiting for the cost to come down, and I need to get comfortable with shipping disk drives around. (Although, tape cartridges are getting more fragile with each redesign.)

And no, Amazon S3 and similar services are not cheap enough yet, both for Amazon's charges and the bandwidth.

Thoughts on cloud storage

2009-12-19T10:12:00.000-08:00

We have hundreds of terabytes of active storage on our servers. Most of that storage is cold storage, so we've focused on inexpensive, reliable, slow, online storage. We use tape for offsite backups only, more on that later.

Lots of noise about cloud storage, so I started running the numbers using Amazon's published numbers.

Two problems popped out immediately:

1. AWS storage is a lot more expensive than doing it our selves, even with some imaginative scenarios.

2. We need a lot more Internet bandwidth if we're going to move multiple terabytes of files between our servers and Amazon's.

As I run across other companies offering cloud storage, I run the numbers again, and pretty much come to the same conclusion. Nobody's undercutting Amazon by much, and Amazon has been dropping prices.

Problem #2 is a problem for everyone, and will continue to be a problem, until another solution is offered. Possible solutions:

We take a "station wagon full of tapes" or disks to an Amazon site for upload to their servers.

We rent fat pipes, space, and power for short terms from a well-connected ISP, so we can truck a server to their location to upload the files.

Cat 6 or 6A cabling?

2009-12-17T13:47:00.000-08:00

The usual dilemma when installing a new network, especially along side an existing network.

Our existing network (about half the total space) uses Cat 6 wiring installed 3-6 years ago, before Cat 6A was widely available. We have Gigabit Ethernet to the desktop. GbE is good enough most of the time, but some heavy data users and some network backups could use more bandwidth.

Which means I should put in Cat 6A in the new space to "future proof" the network. Which costs more. Which might not be in the budget. And we definitely don't have the money to rip and replace the existing Cat 6 cabling, creating bandwidth "haves" and "have nots" in the building.

Cat 6A cabling is much thicker than Cat 6, which will likely cause problems with conduits, ladder racks, cabinet punchouts, bend radii, etc.

By the way, POTS (Plain Old Telephone Service) runs just fine over 2 pairs of Cat 6 cabling terminated in RJ-45 jacks. RJ-11 cables plug right in with no problems. Any jack in any office can carry either POTS voice or Gigabit Ethernet data, with a couple minutes in the telecom closet swapping jumper cables.

We probably won't decide between Cat 6 and Cat 6A until we get cost estimates in a few months.

Four standard answers

2009-12-17T10:19:00.000-08:00

A few years ago, the people I raft with started circulating the Four Standard Answers that may be used by whitewater raft guides in response to nearly any question:

It depends.
About 45 minutes.
Just around the next bend.
Chest high on a duck.

Figuring out what these answers are good for is left as an exercise for the reader.

Especially the last one. Think about it.

No raised floors!

2009-12-16T23:42:00.000-08:00

Been there, done that, have scars, never again.

Traditional data center design uses raised floors, perforated tiles, rows of cabinets surrounded by hulking HVAC units and giant PDUs, and a whole room dedicated to batteries. No thanks. I have a better way.

Raised floors are poor ways to move cold air around, and then we fill the air handling space with power and network cables and wonder why we have hot spots and high fan loads. Raised floors are expensive, dangerous (ever fallen into an open tile?), not very earthquake resistant, and capable of hiding too many sins -- like dust bunnies the size of German Shepherds.

My last few data centers have worked out OK without raised floors. On the other hand, we still have hot spots, visible rat's nests of network and power cables, and the occasional dust bunny. This time will be better.

What's a good data center like without raised floors? Turns out we can steal a few ideas from good old Ma Bell central office designs:

Everything is mounted in cabinets bolted to concrete platforms. The platforms raise all the equipment above the inevitable self-inflicted floods, without enabling the hiding of cables, dirty magazines, and other sins. In earthquake country, we'll put down seismic isolation platforms (like these).
Power and network cables run in overhead trays or ladder racks.
Run with wider temperature ranges, no humidity controls, and lots of fresh air ("air side economization").

Going beyond Ma Bell:

Use overhead three-phase power busses (like these) to maximize power flexibility to each cabinet.
Use 208V single-phase AC power whenever possible to maximize efficiency; 110 VAC if really needed. DC powered servers are still rare, and DC power operating knowledge is rare, too.
Strict cold air/hot air isolation. Flood the room with cold air, force that air to go through IT equipment only with baffles and seals, and vent the hot air through cabinet-mounted chimneys (like these), directly back to the HVAC system or outdoors.
With a 60 KW data center, use cabinet-mounted small UPSes, instead of a central UPS that's typically oversized and running inefficiently.
Keep the HVAC units out of the data center as much as possible. Ideally, roof-mounted units, with big fat air ducts into the data center.
Absolutely No Humidification Water in the Data Center. Rube Goldberg must have been lead designer of all of the humidification systems I've seen. Stopping water leaks and sprays with dozens of kilowatts of power a few inches away is no fun. Can't get rid of water entirely, I always lose the sprinkler arguments with the fire department.

More later ...

Phones are a PITA!

2009-12-16T23:20:00.000-08:00

Seriously, I've suggested ditching all but a handful of land lines since most of our staff have cell phones. Nobody's willing to cut the cord(s) just yet.

Our existing space has about 120 AT&T Centrex lines. Very old school, very cheap to install, very expensive monthly charges, very easy to administer, and the phones work when the power goes out -- which is frequent here. Centrex doesn't have a lot of fancy features, and you pay a lot to add them. We don't need much -- voicemail on about half the lines, a few people need Caller ID, that's about it.

Ideally, we'll add another 120 Centrex lines. Checking with AT&T to see if they have another 120 pairs available here. We're a couple of cable miles from the CO, so I'd be astounded if they pulled more pairs just for us. Last time we added lines here, didn't look like they had a lot more pairs coming into the MPOE, and other tenants have moved into previously-empty spaces since then.

Our only alternative is an Avaya VOIP system. Corporate standard, no other vendors allowed. We'd dump our existing Centrex and go all VOIP, need about 240 lines. Lots of features, sexy new technology, security issues, high up-front cost, high administrative load, need some POTS lines for backup when the power fails.

Still waiting for a quote from our contract, but a pretty good estimate shows the VOIP system will cost about $100K more up front, with a pay back time around 2 years. Regardless of the payback time, I might have a hard fight to get the extra $100K up front.

If we can't get the Centrex lines, the decision is easy.

I'll know more in a few weeks.

It's not about square feet

2009-12-16T22:56:00.000-08:00

For those of you new to data center design: it's not about square feet, or watts per square foot.

It's about total kilowatts, and cubic feet per minute.

Yes, you need a certain number of square feet to hold your equipment cabinets, but probably less than you think with careful planning and looking at all design elements.

Electric power and air are your ultimate limits, friends, and enemies at the same time.

Servers are getting more power hungry. A cabinet full of blade servers can draw 30 KW or more! A 42U cabinet full of 1U servers can total 42 KW.

Virtually all of that electricity is turned into heat. Imagine 42 electric heaters locked into a 2 x 4 x 7 foot cabinet. Gets hot fast.

That's where CFM comes into play. You get that heat out of there by moving lots of air at the right temperature. Forget about water cooling unless you want lots of headaches and design limits. 99.9% of servers use air now and for the foreseeable future -- we just need lots of it.

About 100 CFM per KW more or less. That's for a 30°F temperature rise through your server.

So for my 60 KW data center, I'll need about 6000 CFM of air moving through the servers.

Add 20% for leakage, and I'll need about 7200 CFM.

Without crowding, I can get twelve 42U cabinets into 500 square feet. With a 60 KW limit, that's 5 KW per cabinet. Now you see why it's not about square feet.

60 KW is about four times as much power as we're using now, and our existing data centers total about 1200 square feet. Already we're saving construction and rental costs.

Now, how do we get kilowatts in, and CFM flowing through, reliably and efficiently? Stay tuned.

New space, new data center, new networks, new phones

2009-12-16T22:42:00.000-08:00

We're finally moving the other half of our group into newly remodeled space adjacent to the first half, about 70 people joining 70 others.

I'm designing a new data center (500 sf, 60 KW, not very big), a new network, maybe new WAN links (currently DS-3 45 Mbps), and probably a new phone system for both the new and existing space. We're working through multiple levels of indirection, so this will be challenging.

Good news:

I've done this before, and I like doing this
I've had a couple of years for research and design, due to various delays

Bad news:

Tight budget and deadlines.
Need to minimize operating costs, too
Big "green" push

I've learned a lot in the last couple of years.

Bottom line: like anything else, lots of vendors out there have "solutions" that make them rich and don't meet our needs, while claiming to be "green".

Details to follow.

Introduction

2009-12-11T13:27:00.000-08:00

I know a lot about some topics:

Information Technology - security, storage, networking, data center design
Whitewater rafting
Lightweight backpacking and day hiking
Wilderness first aid

Why should you read what I write?

I've worked in IT and IT management for over 30 years
I've been a whitewater raft guide, guide trainer, and trainer-trainer for over 20 years
I've been backpacking and hiking for over 40 years
I've kept up my wilderness first aid training for over 30 years, WFR certified for 15 years.

And yes, I tend to think and write in outlines.