Working as an operation manager and DBA often requires me to examine and produce statistical reports and devise actions based on them. I had my fair chances at seeing misinterpretations of statistical data resulting in very harmful actions. A former work colleague of mine suggested reading two books: Analytical Thinking and How to Lie with Statistics. Both books were very useful in designing statistical tests, explaining and analyzing statistical reports. It changed the way I look at statistical research, which I encounter in my day to day life. I will try to explain some of the errors in the planning and analyzing of data, which we all encounter in our day to day life.
Often statistics are used for promotion and sales: The ten minute commercial breaks are annoying, and like most people, I usually switch to a different channel when they start. The time span for a single commercial sometimes dictates the content that will promote sales, but might have poor connection to the actual product. Lately, I can see more and more commercials citing statistical information, but if you read the small writing, you will start to wonder..
Statement: “According to a survey, voted the best insurance policy” or “According to a survey, voted the safest car for 2009”. Small letters: “according to public opinion survey taken by….”. Are you kidding me? What does the public know about car safety, or insurance policies (do you understand the small letters in your policy?)? Last time my sunglasses went missing from my bag, the insurance representative told me that since I wasn’t robbed (you know, “put your hands in the air” robbed) I’m not eligible for any compensation and then he pointed to page 3 paragraph 13- section 4!!
Statement: “Voted the best product of the year”… Small letters: “According to a survey taken in 2007 by…“. Are you kidding me? In the last 3 years new companies and technologies have entered the market! Wake up! It’s 2010.
Statement: “Voted the best car 2009”… Small letters: “according to a survey taken in Italy … “. I’m not sure, but the last time I checked in Italy, most cars have a manual gearbox. In Israel, on the other hand, most cars have automatic gearboxes, and, the last time I bought a French car, after three visits to mechanic and some unnecessary oil and part changes, they found that the radiator heat plug did not match the Israeli hot desert weather, but the European cold winter. So guys, please try at least to quote a relevant statistical survey.
When someone like Milla Jovovich is “selling” me that “applying this mascara will increase your eyelash up to 30%” or “using this moisturizer will remove your..”, I ask myself: could being beautiful to begin with has an effect on results of applying cosmetics products?
So what is the relevance of the issues I have mentioned above:
Expertise in your field.
Select relevant data.
Use demographic analysis and slicing.
What comes first? Chicken or the Egg? Select the subject group so it will not affect the result of the test.
In every test you design or analyze, when planning the test you must consider carefully the place and influence of these issues on the test.
In my next post I will try to show you the same issues in your day to day life and the design of your test.