Because of my background in engineering and computer science, I generally care more about numbers than most people.  I don't feel comfortable when decisions are made because someone has a gut feeling.  Gut feelings are normally wrong.

However, using data to influence decision making can end up being even worse.  Having information and doing research is rarely a bad thing, but people tend to think that the results of their research are much more significant than they are.

So first, the obvious example of probability/statistics gone wrong:  Apple's iPhone sales increase by 300% in 2008.  Then some tech reporter writes a story about how Apple is on pace to have the most market share in 2 years.  Linear Regression is not normally a great way to predict the future and it's especially bad when you have a sample size of two.

Sometimes people do a good job of analyzing the first round of data but they ignore opportunity cost.  Just because launching a new website increases traffic doesn't mean that a different website couldn't have increased it even more.  Just because you can normally make money from buying a house and selling it 10 years later doesn't mean that you're making more money that if you rented.

That's just two common examples.  There are all kinds of potential pitfalls if you put all your faith in numbers, so here's how I'm trying to avoid that trap:

• Progressive Enhancement - Sometimes you just don't have enough data to make a decision.  Luckily you can slowly start implementing an idea and collect data as you build it.  Sure, it's better to have numbers to start with, but I'm normally not that lucky.
• Be Pessimistic - I'm a dreamer.  I always think my ideas will work and I balance this by trying to underestimate anything that I don't know for sure when running numbers.  This doesn't exactly make my calculations more accurate, but it generally reduces the consequences if I'm wrong.
• Never Be Satisfied - Very few things can be truly optimized.  Even if a strategy is working, something else might work better so I force myself to try a new approach to everything.  For example, I made a website design that I really liked.  I was completely set on using it but I forced myself come up with another design anyway.  That other design ended up being way better.  Then I made a third which was even better still.  It's never safe to assume you got something right, no matter how successful it is.
• Remove Noise - Every equation has variables that don't matter.  Before offering an opinion, I like to say,  "All else being equal..." but first I make sure I know what "All else" is describing.  At work the other day we were trying to decide between two different projects to work on.  One of the projects can't possibly have an impact until 2010 (for tax reasons).  Once we realized that we were considering things that weren't relevant, the decision was easy.
• Make sure you collect data - One thing that we're great about at work is collecting information that we aren't using.  We have a log of every hit to our site since 2007 even though we didn't start doing anything with that info until this year.  It's a lot easier to use data in your decision making if you thought ahead and started collecting everything possible long before it really matters.  This is why Google is successful.
• Link Data - This is something that Google could do a lot better (at least with the tools they provide to developers).  I use Google Analytics, Webmaster Tools and Feedburner.  Between these services Google could tell me anything I would ever want to know about my website visitors.  Unfortunately they keep all three as separate services which means there's a lot of guesswork on my part.  Our analytics at work are 100% home-grown so we can see all the relationships between emails, newsletter subscribers, visitors, etc.