Musings on IT, data management, whitewater rafting, and backpacking

Tuesday, April 19, 2011

My 2006 IT forecast

In March 2006, I was asked to write an IT forecast for upper management, to guide strategic planning.  I stumbled across that forecast again today, removed some internal references, and posted it below.

To IT people, and in hindsight, these forecasts seem obvious.  But they were not obvious to non-IT upper management at the time.

Consumers will drive most advances in information technology, not businesses or Governments.

Broadly, IT advances have been driven by three waves:

First wave: Government requirements

From the Census of 1890 through the 1970s, most computing advances were driven by Government requirements. Much IT was developed to meet the unique requirements of the Government, then commercialized for use by business.

Second wave: Business requirements

  From the 1980s to the early 2000s, we bought commodity computers that were sold worldwide. These computers and other IT were developed to meet the needs of large businesses. This drove down the cost of IT, but made meeting our unique requirements more difficult in other ways, including increasing security problems. The PC revolution brought widely available computers, networks, and software, so much so that we barely remember Government-unique systems, except in DoD.

Third wave: Consumer requirements

Increasingly, consumers are driving IT requirements. A much larger market than for business or governments, we're seeing costs come down substantially, ease of use increasing, but security, reliability, and maintainability dropping. Cell phones, wireless networks, cheap Internet access, digital cameras, iPods, high-speed graphics cards and personal computers designed for video games, GPS receivers, are all developed for consumers, then re-used by business and the Government. The rate of change of consumer technology is much faster than for the previous waves.

Implications for planning:
  • More than ever, our needs will be met by cobbling together and reusing technologies designed for other purposes. When iPods are used to help identify birds using songs and pictures, blanket policies prohibiting the purchase of “consumer” technologies will hurt our science.
  • Some of our needs (for reliability, security, ...) will be more difficult to meet, and will require new approaches. For example, rather than building one very reliable data acquisition system connected to one reliable network, we might be forced to collect the data several times with cheap devices, send the data several times over a variety of unreliable networks, and figure out which data stream is the “correct” one. And when one of those devices or networks fails, we'll need to be flexible enough to use the “new improved” model that doesn't work exactly the same.
  • Central control of IT will be virtually impossible. As long as scientists can acquire very powerful technology with credit cards, or simply buy very cheap technology out of their own pockets, we must lose any illusion of “controlling” our IT environment. For example, restrictions on Internet access are nearly impossible today – I can access the Internet from my office through at least four completely different wireless networks, three of which have no policy covering my use.

Disk storage per dollar and per disk drive will grow dramatically for the foreseeable future – doubling every 18 months or less.

Implications for planning:
  • We will collect a lot more data a lot cheaper than before. This is a double edged sword – we'll be able to do things we couldn't do before, but we'll be swamped by all the data we can collect. For example, one group has switched from collecting only data from rare, significant events from their worldwide network, to continuous recording, and the research benefits are only starting to be explored. Continuous high-definition video recordings of a variety of natural phenomena are creating whole new research techniques and scientific discoveries.
  • Our data management problems will double every 18 months, and other parts of the system won't be able to keep up. For example, we should be keeping offsite backups of all our data. Tape drive capacity isn't keeping up, and network capacity isn't keeping up. We will see some very large datasets lost to disaster, accident, or deliberate destruction in the near future.
  • We'll need to prioritize what data we keep, and what data we protect, because it will be too easy to “keep everything” but protect and use almost nothing.

Maintaining good computer security will not get easier.

Hackers improve attacks as fast or faster than we improve defenses. People are always the weakest security link, and human nature hasn't improved substantially for a million years. New technologies are developed with little thought to computer security, especially as profit margins decline and consumer ease-of-use (see other IT trend) trumps improved technical security.

Implications for planning:
  • Cooperating with external partners will not get easier, unless we radically rethink how we do either computer security or collaboration.
  • Resources devoted to computer security will not go down, even as our overall budget shrinks.
  • Improving security to meet external mandates will have a major impact on budgets, major operational impacts on collaboration and even using computers to do our science.
  • Scientists will silently subvert computer security policies and technologies in order to get some work done.
  • Insider attacks will increase as our workforce gets more computer literate. We won't be able to use a security model that trusts “us” but doesn't trust “them”, for arbitrary definitions of “us” and “them”.
  • Technical security measures will improve, but improving processes, and changing people's computer security habits, will remain major challenges. When mandatory computer security classes reach 40 hours per year...


Raw computer speeds will increase more slowly than in the past. Multiple-core CPUs, and multiple CPUs per computer, will become the norm.

In a few years, desktop computers might contain the equivalent of eight central processors, each one only a little faster than today's fastest chips.

Implications for planning:
  • Lots of software must be rewritten in order to effectively use multiple CPUs to analyze larger data sets. Old software that's not rewritten will not run substantially faster in the future, and won't be able to handle exploding data set sizes.
  • Software development, acquisition, and maintenance costs could rise substantially.
  • Our programmers and scientist/programmers will need training in new software development techniques, and will need new software tools to write, debug, and maintain software running on multiple CPUs.
  • Our system administrators will need training to effectively run multiple-CPU desktop and server computers.

END OF 2006 FORECASTS


How about the forecasts I didn't make?

I deliberately omitted grid computing, now known as cloud computing, because we had too many internal and external barriers to widespread use. Turns out that was a good call. Except for a few people using consumer clouds like Gmail and Google Docs without support or authorization, we've made almost no use whatever of cloud computing. Most of the barriers are still in place, despite top management pronouncements to “consider clouds first”. And we've added barriers to consumer clouds like Facebook, Twitter, and Dropbox.

No comments:

Post a Comment