Twitter Experiment Fail -> Social Graces Fail


So over the last couple of days, Brian Dunning of Skeptoid has been running an experiment (I think what he meant was “informal survey”) over Twitter consisting of 8 questions. When answered, a tweet is automatically generated that the participant is to copy to their twitter feed.

Enter problems.

People were complaining about the huge amount of spam generated by this “experiment”. While I agree that this was not the best way to do things, I used to be a researcher and am no stranger to the trial-and-error that takes place in experimental design – especially for something relatively informal that does not have a team of experts working on the design. Maybe that’s why I had a little more patience for this and could ignore the extra traffic without too much bother.

And yes, Richard Wiseman did it better. Good for him (seriously). But Brian Dunning is not Richard Wiseman.

Now, I’m not the hugest fan of Mr. Dunning due to some recent comments he’s made, but the subsequent responses to his experiment are starting to become almost hostile. It’s one thing to give constructive criticism so that he can learn and try better next time, it’s another to needlessly make sarcastic remarks like “I’m participating in the Skeptoid Twitter experiment. My answer is [insert non-sequitur snarky response]” and speculate about his motivations. That’s just rude.

The accusations, that are entirely without evidence by the way (who can say they can accurately comment on the internal motivations on another person?), that this is a ploy to get Skeptoid subscribers may be false. Even if true, I don’t think it’s inappropriate for people to advertise something they want to share – especially in this case as the experiment was advertised as being run for Skeptoid in the first place. “Subscribe to Skeptoid to get the results” is another way of saying “I have a podcast that you might like to listen to” and even simply “here is how you can get the results of the experiment you just participated in”. Anyone who doesn’t want to listen because it’s not their thing can easily not subscribe, unsubscribe, or download that one episode. It’s a FREE podcast. Who cares? He’s not getting anyone to buy anything, so what difference does it make?

So, sure. Make your opinions known, give feedback, whatever. But what’s going on now is just silly.

UPDATE: Brian Dunning comments on his Twitter experience this week.


5 responses to “Twitter Experiment Fail -> Social Graces Fail

  1. I agree that many of the responses are hostile (some contain personal insults), which is a shame but he has broken many rules of marketing and social netiquette by making the explanation and results conditional on taking action which directly benefits him. He is gathering data from people who don’t listen to his podcast, via a third party application. He is asking them to give their time. That’s all fine, no different to Wiseman’s experiment, for example. However, the golden rule of asking for data and time is to reward the giver. Instead, he’s asking for something else. He is creating the condition that you can only get an explanation for your participation, and the results, if you listen to his podcast. That is really not acceptable, given his podcast is not the mechanism by which he collected the data.

    If he had asked his podcast listeners for the data only, then fine. But he’s asking the web in general, and twitterers in general, for the data. That creates a social obligation to publish an explanation and the results in the same arenas (i.e. the website and/or twitter), not exclusively on his podcast.

    You must remember that the podcast is not a factor in the experiment (or at least it might not be. I’m not allowed to know what the expt is all about unless I listen to the podcast). So the only motivation for restricting certain information to the podcast is cynical promotion, surely? What other motivation can there be?

    Plus, listening to podcasts is not the same as reading a webpage. It is an additional obligation. It requires additional tools from the user, and extra time to download, extra clicks to get to. Again, this is considered rude at least by my standards of voluntary participation.

    • I think his major mistake was referring to this as an experiment. As far as I know he is not a scientist, and this is not a scientifically rigorous experiment. It is more of a survey. He publishes each episode he does on the web as a written version. But either way, it’s a free podcast. He probably could have been more open about the various methods people can access the information, but I take no issue with him promoting something that is free. It doesn’t obligate them to do anything any more than they were obligated to participate in the first place. If they had issues with his methods, they could simply ignore the whole thing.

  2. Note to my comment above. I’m basing my opinion on the wording of

    Which states “What’s it all about, and what are the results? Subscribe to the Skeptoid podcast to find out.”

    This is what I have a problem with, because I absolutely shouldn’t have to listen to or subscribe to a podcast to get the results of a survey I participated in online. But if he publishes the results in a non-podcast format as well, then I have no problem with it and wish him well in his experiment.

    • As far as I know, he’s not super experienced with this kind of thing. So hopefully he learns for next time and does things with a little more planning.

      Again, I agree that this was not the *best* way to do things, but I do feel that some people were being extremely harsh and that was the main motivation behind me speaking up here.

  3. Also, to EVERYONE reading this: this is NOT an experiment by the strictest sense of the term. There is no REB, there is no statement of privacy or participation – all of the things that come with a real experiment. This is a SURVEY. So technically he can do what he wants, whether or not it’s the best method.

    Go ahead and give him feedback – that’s entirely appropriate. What I don’t agree with is the snarky nonsense that was going on over Twitter earlier, which seems to have died down a bit.