You may have seen Shaan Hurley and I having a discussion (ahem) over the validity of his statement:

I really do use the ribbon now with AutoCAD 2010 along with most users as evidenced by the CIP data we receive daily from thousands of AutoCAD users who choose to send the great data.

So, now you know. Most of you use the Ribbon now,  Shaan said so. Shaan, as he always has done in the past, declined my invitation to back up this assertion with more details. He has vast amounts of data collected from huge numbers of users. How could that possibly be wrong?

Here’s how. CIP data is biased.

How can millions of data points be biased? Actually, all samples are biased. Only the degree of bias varies. The polls on this blog are no exception. I do my best to keep the questions and options neutral; the only leading questions you’ll see here in serious polls are the ones I copy and paste from Autodesk blogs. But readers of this blog are one self-selecting small portion of Autodesk customers, and people who vote in my polls represent another self-selecting portion of that portion.

The question is, how biased is Autodesk’s CIP data? Without access to Autodesk’s data (which it won’t provide) and resources for alternative data collection from its customers (ditto), the best I can do is use my own biased sample (that’s you lot out there) as a cross-check.

Let’s examine it in light of Ribbon use among AutoCAD 2010 users. In an earlier comparison of my 2009 poll figures and Shaan’s CIP data, I wrote this:

But Shaan’s CIP users are also a biased sample, comprising those AutoCAD users who have CIP turned on. Are users who go with the flow and have CIP on also more likely to go with the flow and leave the Ribbon on? Possibly, but I would have thought the CIP-on bias would be less significant than the blog-reader bias.

I have recently run a poll to try to determine if that “possibly”, that hunch, has any basis. Let’s examine the results I got.

AutoCAD 2010 users, what are your Ribbon and CIP settings?

Ribbon on, CIP on (24.7%, 65 Votes)
Ribbon on, CIP off (19.4%, 51 Votes)
Ribbon off, CIP on (11%, 29 Votes)
Ribbon off, CIP off (44.9%, 118 Votes)

Total Voters: 263

For the sake of argument, let’s make the assumption that my poll sample is unbiased. It’s not, and the degree of bias is unknown, but let’s see what it would mean if it was. Let’s see what kind of results Autodesk would see from its CIP sample:

CIP-on voters (94):
Ribbon on 69% (65)
Ribbon off 31% (29)

Shaan would see from this result that 69% of AutoCAD 2010 users have the Ribbon on, and would be tempted to say stuff like “use the ribbon now with AutoCAD 2010 along with most users”. Understandable. That’s just CIP users, but non-CIP users can’t be that different, surely? Or can they?

CIP-off voters (169):
Ribbon on 30% (51)
Ribbon off 70% (118)

Wow. That’s a huge discrepancy, and it implies that a sample selection based on CIP use introduces a massive bias. I’ve watched this poll grow over the weeks, half-expecting things to even out as the sample size increased. It didn’t. It has been pretty constant, with non-Ribbon non-CIP users outnumbering Ribbonite non-CIP users by a substantial margin.

Let’s put the groups together, shall we?

All voters (263)
Ribbon on 44% (116)
Ribbon off 56% (147)

So, if the voters in my poll were observed by Autodesk via CIP they would appear to be 69% Ribbon users. In fact, only 44% of these voters are Ribbon users.

How many AutoCAD 2010 users really have the Ribbon on? 69%? 30%? 44%? Some other number? I don’t know, and that’s not the point. The point is, Autodesk doesn’t know either. It can take some smart guesses, but just assuming CIP is accurate isn’t smart, it’s just a guess.

Why does this matter? Because Autodesk makes decisions based on this stuff. Decisions that affect you and me and how we use our tools. Have a look at this statement from Autodesk’s Teresa Anania, Director of Industry Management (taken from her interview with Deelip Menezes about Inventor):

…we had data that suggested that the new ribbon UI was well received and would be absolutely all that customers needed …. And now since we have the CIP data that shows us how our customers are using the software, we can analyze this before we permanently turn anything off.

Comments like this (and others from other Adeskers) seem to indicate that there is an unspoken assumption that CIP users accurately represent a true cross-section of users in general.

I know that Autodesk doesn’t rely solely on CIP; it uses a wide range of research tools to find out what users are up to and what they need. I regularly encourage you to participate in various Autodesk surveys, for example. But there are problems of accuracy inherent in all those methods. It would be natural, when faced with a set of apparently conflicting results from different sources, for Autodesk decision-makers to simply assume that the source with the biggest sample size is the most accurate. That could be a dangerous mistake, for both Autodesk and its customers.

Note: my arithmetic was off in several places when I posted this, and I have edited the post to correct some of the figures. These corrections do not invalidate the arguments; the substantial bias is still evident.