What did people crowdsource with MobileWorks in 2011?

Here at MobileWorks, the next-generation replacement for Amazon's Mechanical Turk, we've processed nearly half a million crowdsourcing and human computation tasks since launching in mid-2011. 

We've built a platform that's powerful and versatile, but we're still constantly surprised by the ingenious uses of human brainpower that our customers come up with. 

Picture_81

So, what did people crowdsource through MobileWorks in 2011? Here's a breakdown of the 7 most popular and most innovative tasks we saw:

#1: Human-powered search and information collection

Organizing the world's information constituted nearly 30% of our brainpower usage this past year. While modern search engines do a great job at gathering the individual pages you want, they can't yet deliver the exact information you need.

What do you do if you want to find the homepages of 1000 people working as software engineers and willing to move across the country to take a new job? How about if you want to generate an up-to-the-minute database of the most popular products sold on Craigslist in San Francisco? What if you need to generate a verified list of leads for a sales team, like 100,000 restaurant phone numbers in New York City? 

In these situations, Google may be able to deliver individual pages for you, but it isn't able to extract and collect exactly what you're looking for. A trained crowd is still the best way to get it done.

We aren't surprised that this task is so popular, as the ability to quickly gather this sort of information is enormously valuable for recruiting, lead generation, and maintaining databases. 

(If you're interested, MobileWorks has a dedicated application for web scraping: the Excavator.)

#2: Text extraction: OCR, handwriting recognition, form digitization & data entry

Digitization exploded in 2011, fueled in part by the medical industry's push to move paper records to digital ones.

Why use the crowd for a classic AI problem like OCR?  Although plenty of software-based OCR solutions exist, they're often expensive and aren't great at understanding handwriting that's a little sloppy or unusual. They trip up on documents with mixed typefaces or changing orientations. Simply put, they can't handle corner cases. Crowds provide a completeness and robustness that traditional OCR doesn't.

On top of that, lab research shows us that crowd solutions can digitize documents that are simply beyond the capabilities of OCR software. In one experiment by our friends at MIT, a crowd deciphered this handwritten note:

Ocr
image credit: Greg Little, Lydia B. Chilton, Robert C. Miller, and Max Goldman, HCOMP 2010 

You misspelled several words. Please spellcheck your work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney. You do make some good points, but they got lost amidst the writing. (signature)

Let's see OCR do that!

In 2012, we expect to see more and more companies with piles of paper choosing to send their work to the MobileWorks crowd to be digitized, rather than investing in developers or OCR software solutions.

#3: Tagging and sorting

As the web produces more user-generated content is produced -- photos, podcasts, and more -- more of it needs to be tagged by humans. Don't just think of your pictures on Flickr and Facebook: MobileWorks received requests to tag user-generated content in games, relevant segments in videos, and customer phone calls for automatic follow-up.

#4: Content filtering

Was that post naughty or nice?  Machines do a good job keeping pill ads out of your email, but for almost all other kinds of content – marketplace listings, advertisements, and more – humans beat machines every time in checking whether content is relevant, interesting, and on-topic.  More companies are finding that it's simpler to use a one-line crowd solution than try to fight a technological battle against spam.

#5: Advertising analysis

The popularity of ad-distribution networks sometimes means that advertisers don't know where their ads end up on the web. It's critical to understand the content of an ad and an article to avoid slip-ups like this:

Badad

Image credits:  The Stupidity of Scripts, James Duncan, Flickr

We had two common advertising-related tasks: judging which company was behind a particular advertisement, and judging whether an ad was appropriate on a certain page. The cost of slip-ups are in ad placement are fairly high for advertisers and content producers alike, and it's very simple for humans to determine whether a particular combination is appropriate or inappropriate.

#6 Image recognition and description

We saw a good number of tasks that involved recognizing objects in images, classifying objects in pictures, or tagging images for a machine learning system to analyze later. These tasks came from a variety of sources, including a number of mobile phone applications letting users ask questions about documents or objects in the real world seen through their cameras.

This is another application where lab research paved the way: Rochester's VizWiz project allowed visually-impared people to upload pictures from their mobile phones and get descriptions of what they photographed back in seconds.

#7 Audio transcription and analysis:

Whenever a podcast is produced or an interview goes up on a tube site, advertisers want to be able to place relevant ads in it. And until speech recognition can deal with complex words, music, and accents, it's still better to use qualified humans.

And, without further ado, 2011's Task Of The Year...

Human powered website testing took the role as the most interesting task we saw in 2011.

It's sometimes difficult when maintaining large projects to write test cases that simulate extended human interaction with all parts of a website. 

But it's dead simple to ask a human on MobileWorks to carry out a list of tasks on your site and make sure everything's working OK. It's cheap that many developers wanted to include a call to MobileWorks in their "automated" test routines, as a final deployment step.

What do you want to see crowdsourced in 2012?

Crowdsourcing is growing by leaps and bounds, and we can't wait to see what the next year will bring.

If you've got an idea for something you want to crowdsource, what are you waiting for? There's no easier way to crowdsource work than MobileWorks.  Try it today for free!

Filed under  //   2011   crowdsourcing    human computation   mobileworks   search   state-of-the-crowd-report   tagging   tasks  
 

Turbocharging MobileWorks: Job management, webhooks+callbacks, and HTML5 support

Since launching our beta in August, we've received a lot of feedback from developers about how to make MobileWorks better, faster, and stronger. We've launched a new generation of features that make using crowds in your applications easier than ever. Here's a quick look at what's new:

HTML5 Support: 

Designing crowdsourcing tasks has historically been a major difficulty for developers, so MobileWorks has always made posting simple tasks really easy. Here's an example of how a question is presented to the crowd:

System-screenshot

Almost all of this rendering and design happens without any action on the part of the user – you just ask your question and it's presented naturally.  

However, a few developers had needs for more sophisticated, customized interfaces for interacting with the MobileWorks crowd.

As a result MobileWorks now lets you embed complex questions into a Javascript or rich HTML interface that lets you post any structure you like. It's as simple as putting HTML and JS into your 'question' parameter.  Be as creative as you like!

Job Management:

With bulk uploads, developers can now post hundreds of task at once instead of having to loop over individual tasks. A bulk upload unit is called a "job" and can consist of any number of individual tasks.  An example of a curl call is given below:

curl --dump-header - -H "Content-Type: application/json" -X POST --data 
'{"tasks": [{"question": "What is two plus two?", "answerType": "t"},{"question": "What is seven plus eight?", "answerType": "t"}],
"webHooksUrl":"http://www.yourURLhere.com/"}' https://sandbox.mobileworks.com/api/v1/job/ -u yourusername:yourpassword

New API options: 

trackingId lets you, optionally, mark individual tasks with a unique tag of your choice, for better integration with your own task management platforms. 

webHooksUrl: Webhooks let you receive the answers for a job as soon as they're available via a POST request to a url of your choice. You don't have to poll tasks for answers anymore. You can receive callbacks for individual tasks as well – just post a job with a single task and you'll receive a callback as soon as the task is done. 

Want to get up to speed fast? Just visit http://www.mobileworks.com/developers/index.html

Try it now!

Here at MobileWorks, we're always working on improving giving our developers the right tools to quickly integrate crowds into their application, as we build the world's best crowdsourcing engine. Try it out and let us know what you think.

 

 

MobileWorks in the Press: October Round-up

Crowdsourcing makes for a spectacular tech press story, and MobileWorks is no exception. We combine something fundamentally exciting and new from a technology standpoint – reliable API access to human brainpower! – with a human interest story that can't be beat.  After all, where else can you solve hard AI problems while battling poverty?

In a recent Xconomy interview, we talked with Wade Roush about how we expect technology crowdsourcing to provide a new kind of employment option to folks out of work both in the US and throughout the world.

MIT Technology Review writes about our upcoming native support for real-time crowd applications, mirroring innovations happening in the academic research world.

John Roach at MSNBC points out that we're weaving crowdsourcing into computer software for only pennies per application.

We've also seen coverage in Slashdot, an extensive Q&A with the developer community at Hacker News, and launch coverage at TechCrunch

Wondering what all the press is about?  Try MobileWorks yourself online at www.mobileworks.com. It's the easiest (and the fairest!) way to put a crowd into your software.

 

MobileWorks has three new partners.

It's out of the bag. MobileWorks is happy to announce that we've accepted investments from Y Combinator, SV Angel, and DST. This means that some of the smartest incubation money in the Valley is betting that crowdsourcing-as-a-service can follow in the footsteps of powerful technology platforms and marketplaces like Heroku, Dropbox, Clustrix, and AirBnB. 

We're pleased to welcome these three strong groups onto our team as we continue on our quest to change the way the world works online. 

Crowdsourcing is a powerful force and growing trend in the way work is being done on the web. We're seeing new applications emerge every day. Until now, using crowds has required tremendous expertise in how to recruit workers, manage crowds, and obtain accurate results.  By managing these automatically, MobileWorks provides a radically simple way to use crowds in all kinds of applications.

If you haven't tried it yet, what are you waiting for?  You can put human intelligence into your software.  (Really!)  Try it out at http://mobileworks.com.  We're changing the way the world works, for the better.

 

MobileWorks: A new kind of crowdsourcing company.

Hi!  We're MobileWorks, a different kind of crowdsourcing company.  

MobileWorks is based on the simple premise that making crowd workers happy with their work leads to better quality results – especially when combined with a slew of mathematical tactics, clever algorithms, and UX innovations from one of the world's best teams in the science of crowds.

Can we make using crowds dramatically easier than it is today?  The answer is "yes".

Join us as we reinvent the world of crowdsourcing.

About

Crowdsourcing made easy.