Some Things Shouldn’t Be Crowdsourced

by Julie Gomoll on August 30, 2009

Crowdsourcing: The Good

linux
Crowdsourcing has been behind some truly great products and initiatives. Wikipedia and Linux are perhaps the best-known, and there are indeed stellar examples of the power of this model.

Other great examples include:

  • The Netflix Prize for a 10% improvement on their recommendation engine.
  • reCaptcha and their innovative use of the millions of typed words to help digitize old books.
  • Foldit invites people to play a game that ultimately results in new protein-folding strategies.
  • Microlending companies like Kiva are pooling small amounts of money and loaning it to poor business owners with enouraging results.

Opinions can do well as crowdsourced projects, although I contend people are more likely to supply them when they are bitching rather than raving. Still, reviews and other opinions certainly provide value, and I admit I readily use them when deciding on restaurants, books, and products. Companies like Yelp, Amazon, and Ebay are learning to navigate the intricacies (and legalities) of the relationships between businesses and reviews, making the content even more valuable in the process.

What do all these good examples have in common?

  1. wikipediaLogoThere is governance. Although anyone can contribute to Wikipedia, all contributors are not equal. A hierarchy of editors ensures that spam and misinformation are quickly weeded out. It’d be nice to think it’s all perfectly democratic, but it’s not, and it wouldn’t work if it were.
  2. Crowds are used to solve problems or to offer commentary, not to determine strategy or any decide critical business issues.

There’s gray area, of course. Most software developers actively solicit feedback for new features and general improvements. Notice, though, that they rarely promise the most popular suggestion will be implemented. The people responsible for the business strategy are wisely learning what their customers want, but they know better than to leave the final decisions to the public. The customer may be good-intentioned, but they just don’t know enough to always be right.

For some excellent crowdsourcing stories and analysis, be sure to read Wikinomics: How Mass Collaboration Changes Everything by Don Tapscott and Anthony D. Williams.

Crowdsourcing: The Bad

LogoProcessLogos. I cringe every time I hear someone brag that they crowdsourced their logo. For one thing I have an intense distaste for doing work on spec. Most logo crowdsourcing sites (as well as contests) do just this — a bunch of designers create a logo for free, getting paid only if the company likes and select it. But logos are critical to the identity of a company. It’s not just a matter of what looks nice on a business card and a website banner. The always-meaty and useful Smashing Magazine ran an article called Vital Tips for Effective Logo Design, accompanied by this graphic. Do you really want to entrust this critical piece of your image to starving artists out there eager to get their hands on your $200? Pay someone what they’re worth and get it done right.

Strategy. No one knows your business as well as you. Get Satisfaction and UserVoice are great for site-based feedback and ideating. Customers/readers make suggestions and vote on their importance, and the companies do with those suggestions what they will. Austin’s own BountyStorm lets companies post questions with a bounty (usually in the $5 – $15 range) for the best idea. Some questions are perfect for this model — creative Valentine’s Day ideas, for example. Some are a bit worrisome — ideas for a first tattoo (Yikes! Isn’t that supposed to be a deeply personal decision?). And some just don’t belong there. There are a ton of people asking for business names, taglines, and marketing strategies. Do they really think they’re going to get what they need from a stranger? For $10?

Crowdsourcing SXSW: The Ugly

sxsw2010logoAnd now we come to the issue that drove me to write this post in the first place. Programming at paid conferences should not be crowdsourced. Not even part of it. This will be the 6th year SXSW is letting the general public vote on panels, and the 6th year that attendee complaints have swelled about the quality and selection of that programming. Lots of folks come to town and don’t even bother attending the conference, opting instead for the appropriately crowdsourced BarCamp Austin. Last year at SXSWi some really great panel ideas with excellent panelists never made the cut, while panels presented by often pathetically underprepared heavy hitters filled the schedule. The panel-picker was at least a little easier this year, with the option to vote thumbs up or down rather than assigning a number of stars as a rating.

This approach sounds really good in theory, but what ended up happening last year was would-be panelists filled our Twitter streams with panel pimping promotions and microcelebrity popularity contests, while many deserving panelists truly working at the edge of our industry were left in the dust. Bloggers with 10s or 100s of 1000s of readers ready and willing to vote for their panels crowded out those doing interesting, paradigm-changing work. SXSW is too important to the industry to run it as a popularity contest.

I’m not sure what the best solution is, but I have a few ideas that would improve the SXSW experience..

  • Make it clear that there is a committee that is ultimately responsible for the programming. Not just 40% of it. All of it. There should be 2 (preferably divergent) experts in each area of content (coding, architecture, marketing, social media, business, etc.). The committee could be nominated by the public, and chosen by SXSW staff. Or nominated by staff and chosen by the public. Continue to accept panel suggestions from the public, but leave the deciding to the committee.
  • Create a SXSW advisory group committed to uncovering and featuring new voices and industry trends. Some of the A-Listers are fantastic presenters. I’m in no way saying they shouldn’t be on the schedule. But there are some amazing thinkers doing cutting-edge work right here in Central Texas. Commit to finding them. Ask the public for suggestions of inspiring speakers they’ve seen in other settings and personally encourage them to submit a panel.

This year’s pimping wasn’t nearly as bad as last year, due in part to a bit of a backlash to the whole system, as you can see by the following Tweets.

MarlaErwinTweet

CodyMarxBaileyTweet

What do you think? Is it time for SXSW to change the way they manage programming? How do you think it should be done?

Tagged as: , , , , , , , , , , , , , , , , , , , ,

  • Prentiss Riddle

    According to the panel pimper, uh, picker site the crowd is only responsible for 30% of the decision:

    http://panelpicker.sxsw.com/img/sxsw/panelpicker-formula.gif

    Combine that with the asymmetrical flow of information in the process (presumably the staff and advisory board get to see the panel picker results before they decide where to put their votes) and it means that the public input counts for a lot less than one might imagine.

    What I really think the panel picker is for is not crowdsourcing but viral marketing. Every “vote for me!” is a free ad for SXSW. As such, I’d say the panel picker is a roaring (if occasionally spammy) success.

    • Julie Gomoll

      I’d have to agree that it’s a success in terms of marketing the conference, but I don’t think it’s been successful in generating great programming. There’s lots of duplication, too many of the usual suspects presenting, and (seemingly) no attention paid to the presentation skills of the panelists.

      Even though the crowdsourcing piece of this is only 30%, the perception is that it’s very important. Those of us who have submitted panels are put in the awkward position of deciding to pimp it ourselves and risk the ire of followers, or to not pimp it and risk exclusion.

  • http://npirl.blogspot.com Bettina Tizzy

    AMEN. The Panel Picker was a good idea in the early beginnings to get the word out but SXSW is already a crushing scene and doesn’t need to sacrifice quality panels AND obligate otherwise busy people to vote for their friends who must embarassingly pimp themselves.

    This is a win win win prop.
    .-= Bettina Tizzy´s last blog ..This is it! New virtual world Blue Mars rolls out Open Beta on Sept 2 – Plus, news for registered users =-.

  • http://getify.tw Kyle Simpson

    Speaking as one of those “small people” who doesn’t have a huge cult following that I can “crowd” with, I still feel better about my chances of getting my talk proposal picked if I had a chance to get support, than if some small oligarchy controlled everything, and I felt like behind closed doors they probably just only picked the big popular names anyway.

    I may not get picked, but at least I know I tried and I had a shot. The way your proposing, I’d feel much more chagrin at a selection process which was far too opaque to approach fairness.

    It may not be perfect (and it probably does need help), but at least it gives the little guys a shot at generating buzz.
    .-= Kyle Simpson´s last blog ..getify: Naysayers, I AM a "small person" asking 4 votes for my SXSW talk on UI architecture: http://bit.ly/LWyig (cc @chilkari @juliegomoll @tori) =-.

  • http://firecatstudio.com Susan Price

    What a great conversation this is. As involved as some of us in the social media scene, the SxSW panel-pimping posts become deafening and (I can’t believe I’m about to use this word) unseemly. The pimping process itself gives me a negative brand experience for all the speakers pimping and for SxSW. That can’t be good.

    I agree with you, Julie, we’ve seen a decline in panel value overall, and speaker prep specifically, as well as a drastic upswing in pimping. They’re not necessarily related, though. I find myself moved by Kyle’s plea to maintain transparency in the process.

    Can we propose an antipimping, or controlled pimping process that:

    * Improves the signal-to-noise ratio in social media and other communications about proposed panels

    * Doesn’t interfere with panels being spontaneously organized in social mediasphere

    * Rewards panelists for the reduction in (egregious) pimpage

    ?

    The crowd IS trying to do this, with posts like Marla Erwin and Cody Marx Bailey have contributed. And here, in this post and comments. I think the best solution hasn’t emerged yet.
    .-= Susan Price´s last undefined ..If you register your site for free at =-.

    • http://julessays.com Julie Gomoll

      Susan, I totally agree that transparency is critical to the process, but I don’t see why a programming committee couldn’t be transparent. They could even publish their criteria and grading for each potential panel, creating a grading system that includes:
      * subject matter relevance/timliness
      * unique approach
      * speaker experience
      * cutting edge-ness

  • http://laurapthomas.x.iabc.com Laura P Thomas

    I’ve got mixed feelings on this one.

    Yes, there are those I get tired of seeing pimp their panels in Twitter; but, in the reverse there are people I would like to hear from who’s panels I might never discover in the labyrinth of the picker if I didn’t see them mention it.

    The thing this year that struck me as taking it too far was when I saw a Facebook add asking for votes: http://www.twitpic.com/f85ua

    Something’s just not right about that…

    • http://julessays.com Julie Gomoll

      Yeah, I have to admit I have mixed feelings too. I’m tired of the pimping, and I’m loathe to do it myself. I have a panel in the running, and have only tweeted about it once. The feedback was enough to make me not want to do it again.

      I do agree that the crowdsourcing aspect of SXSW is good *in theory*. I just don’t think it results in the best programming, and it most definitely results in some crabby friends ;-)
      .-= Julie Gomoll´s last blog ..Some Things Shouldn’t Be Crowdsourced =-.

  • http://www.unionstreetdesign.com Jeanne Gomoll

    WisCon is a feminist science fiction convention that will celebrate its 34th birthday on Memorial Day weekend in 2010. We’ve been using a sort of modified cloudsourcing method of program design for the past 13 year.

    http://www.wiscon.info

    I think we’ve solved some programming problems with our process, but still need to find solutions for a couple others.

    Like SXSW, WisCon invites everyone to submit program ideas. We’ve built a powerful web-based system to accept ideas, assign people to programs and set up the schedule for our 4-day event.

    Unlike SXSW, it sounds like WisCon is a much more program-centric convention. Hallways are nearly empty during program events, but fill up with crowds between program time slots. This is not to say it is an ultra-serious convention. There are some academic programs, but there’s lots of humor and laughter too. Parties run till the early hours of the morning. Less music than SXSW though.

    In WisCon’s early years, most program ideas were generated by committee members, which was appropriate because WisCon’s emphasis on feminist SF was at that time (late 70s and early 80s) completely unique among science fiction conventions. The passion of the convention’s founders was the thing that made WisCon successful and created a community around it that flourishes to this day. Many of us who were there at the beginning of WisCon 34 years ago are still involved today … and that’s been a good thing for lots of reasons. But if the original group of people had continued to dominate program design through all those years, WisCon’s programming would by now have slogged into an uninspiring rut. Lots of points of view and an open door for current controversies and interests is the key to keeping programming relevant and dynamic.

    Several program committees had guided WisCon programming through 1995, but in that year we decided to make a big change: we invited everyone and anyone to submit ideas for programs at WisCon 20. We received hundreds of ideas that year and have every year since then. We designed an easy-to-use on-line program submission form and sign-up system (not to say that was easy to do!). Ideas are contributed by past attendees, current attendees and people who hope to attend WisCon someday. The panel ideas are often inspired by a favorite stfnal idea, issue or author. Discussions that are exploding on the web (like the 2008-09 RaceFail discussion) come to us as ideas for panels from people who want to continue the discussion with one another in person. The ideas we receive are incredibly diverse within WisCon’s focus on issues of feminism, gender, race and class and their intersection in the world of science fiction and fantasy. Since many professional authors and editors attend, we also do a lot of workshops for writers and editors, and also feature dozens of author readings.

    After the program ideas have been collected, we have to narrow down the number of programs to the ones we will actually schedule. Usually that means separating out a mere 200 or so program ideas from 500 or more submissions. Some of the pruning is done by the program committee members who combine similar panels or delete inappropriate ones, but most of the filtering work is done by the program participants themselves. Not everyone can participate in this stage of program development — just those who are interested in participating as panelists, presenters or readers. We send a link to all potential programming participants — basically everyone who participated in the previous year’s program plus anyone who let us know that they would like to participate.

    Potential program participants go their personal program page on the web and check the program ideas that most attract them. They can say “Please!” to a limited number of programs and “Yes” or “Maybe” to any number of others. We use that data in several ways. First of all, and most obviously, we try to make sure that as many panels as possible are populated with panelists who are really eager to be on them. (The goal is to avoid the dread statement at the beginning of a panel: “I don’t know why I was put on this panel.”) Secondly, we have discovered that this data can also be used to predict the popularity of a panel. The more people who sign up to be ON it, the more people who will want to attend it. We can use the data to decide what size room to use for a panel, based on this popularity rating (adjusted, of course, when a Big Name is one of the panelists). Thirdly, we use the data to prune the program list. If a program idea cannot attract 5 or 6 people as panelists, we know that the audience will be similarly small.

    Sometimes a program idea gets submitted over and over again, year and again. The person who submits the idea sometimes gets upset that we seem to be ignoring their suggestion. We suggest that they consider signing up for the program themselves and try to recruit others to sign up too.

    Running programming using this very democratic method has worked well for us. People tend to attend WisCon FOR the programs not in spite of them, as I see happening at so many other conventions. A huge proportion of our attendees (probably 40%+) is involved in programming as panelists and presenters, in addition to attending. The most frequent complaint we get is about WisCon’s multiple tracks of programs, from people frustrated that they are unable to attend all the program events that they would wish to. To that, we say “you’re welcome.”

    One of the problems for which we are still looking for a solution has to do with choosing panelists from among the list of those who want to participate, especially for those programs that are most popular and attract 20-30 people who say “Please!” on their program sign-up forms. Often, the program committee is familiar with many of the people who sign up and we can choose the most qualified and potentially interesting panelists. But we know we are missing people who have a lot to offer and are new, or are simply unknown to us. And we know that we sometimes choose the wrong person to moderate panels. We are now working on ways for potential program participants to give us some useful information about themselves … without scaring others away by seeming to ask for “credentials.” Oftentimes the best person for a panel is not a famous author or scholar, but simply the person who is most enthusiastic and well-read on an author or stfnal idea.