How do people get their facts?
Read the article and then come back.
NPD and comScore show Apple at around 30% while actual data points towards 50%. That’s actually a big difference. Because comScore put Android at 51% and NPD put it at 61%. Where did they go wrong?
Well, it turns out the much trusted NPD uses surveys to collect data. It makes me laugh. Just like Nielsen and pretty much most market research. In fact, the most common form of market research is in the form of surveys. Focus groups are kind one kind of survey.
What’s a survey? I got this definition from the New Oxford Dictionary:
1 a general view, examination, or description of someone or something:
And here’s a definition from our much loved Wikipedia:
Statistical survey, a method for collecting quantitative information about items in a population
I took a basic Stats course and from what I learned, the larger your sample size, the more accurate your results will be. It’s all scientific, really. ”Scientific”. You see, in science, you do a bunch of experiments and record all the data. The more data you get, the more accurate your information will be. You see, there’s a lot of variables that could go wrong and to be really sure, you have to do it a 1000 times (figuratively speaking, of course. You probably have to do way more).
The biggest problem with Stat, in my opinion, isn’t how well the survey and statistical analysis is carried out. I’m sure people in marketing have that all packed down. I think the biggest problem lies with the source of the data. My audio mentor would always say, “it can only be as good as the source”.
What’s wrong with the source of these surveys? Well, for one, people. I don’t mean to be mean to people (myself, included) but we aren’t exactly the most accurate beings. We forget things. Our memory is funky. There’s selective memory and a few other memory concepts I’ve forgotten. Your brain will heighten certain facts, forget others, and add things. Why? Sometimes it’s because your brain is trying to make sense of it all. It’s new stuff, after all.
So how well do you remember what you bought 3 months ago? When was the last time you bought your phone? How often do you use Facebook every day? I mean, the last question is pretty good. Someone might say 10 minutes but in fact, it’s more like an hour because they go Facebook binging on the weekends after they party.
That’s just one issue. There’s an even bigger issue at hand. Motive. Why would you want to spend 5 minutes (or more) filling out a dumb survey? It’s not like it benefits you. So usually it’s in the form of money. That can hurt the integrity of surveys. If people are filling out the survey because they want money, how accurate can it be? Since there are no right answers, wouldn’t it be easier to just quickly hit the “next” button? I know I did. I would fill out a few surveys a day to win scholarship money (never did win any) and the first few responses were the only remotely truthful answers there were.
The last issue is about selection. However, marketing people will say that they try to be as random as possible when selecting candidates. Although it may be true that they do succeed in grabbing candidates with different ethnicities who live in a variety of states, there’s still the issue of which candidates will volunteer themselves to this survey. This ties in with the last point. Motivation. So unfortunately, the marketing people will always end up with the same people who are only there for the money, be them old or young.
I’ve been very against statistics because many treat it as a science. It’s not. Well, in a way it is. The theories, the formulas, the chi-squares… they are scientific and they are proven in those regards. It’s just the source that never is as accurate. It makes me laugh when I see stats like 61.2%. Really? Your buddy over there got 48.9%, where’s the degree of error? Usually it’s only the last significant digit that is off, but most stats are off by the first digit. It’s just embarrassing.
The data isn’t accurate and it sure as hell ain’t precise to more than 2 significant digits. Because we’re human. And humans don’t have more than a 3-point scale (some say 5).
Anyways, I do have a way to fix up these bogus stats. Why don’t we get the data from the source? You know, from actual sales numbers? Instead of surveying 10,000 people what they bought in the last 3 months, why don’t we aggregate inventory sell-through for that time frame? Instead of surveying random students how often they use Facebook, why don’t you just ask your kid? I kid, but they could just track it with code, which I’m sure they do already.
We need to stop using people as good sources of data. It’s honestly the most unreliable source of information. And if you think I’m just being mean, think of the last major purchase you made and what day it was. Now look back at your credit card bills (online, preferably) and tell me if the dates match up even remotely close. Now repeat with other people. If the number is higher than even 20%, you couldn’t use any of the information. Hell, 5% is already a bit high.
