“It’s little effort to collect the data and to send it to Mighty AI. Otherwise, [we’re] managing large projects and too much manual time.”
Moz is a marketing software and resources company that “develops inbound marketing software, provides robust APIs for link data and social influence, and hosts the web’s most vibrant community of online marketers.” Marketers look to Moz for today’s most advanced and innovative marketing tools.
Moz is a private, Seattle-based company.
The data science team at Moz set out to build machine learning algorithms that extract structured information—specifically, author information—from web pages. The algorithms would enable a larger tool “that analyzes news articles, blog posts and other content to help marketers audit and discover relevant content.” Moz needed high-quality, labeled data to develop, test, and validate these algorithms.
In the past, Moz had sourced this kind of data with manual and/or semi-automatic internal processes. They found that with this project, however, they couldn’t acquire the info they needed with the methods they’d relied on in the past — it was the first time they found they needed an external solution.
The Move to Mighty AI
Upon excluding an in-house process for the project, Moz began to evaluate outside tools. “We ruled out Mechanical Turk because of the amount of project management that would be required on our end,” explained Matt Peters, Director of Data Science at Moz. Other solutions were explored, but the Moz team felt they would require too much heavy lifting, and would not be cost effective. Moz particularly liked Mighty AI’s “stringent quality control measures” and “felt confident we’d get back high quality data,” said Peters.
Deciding to work with Mighty AI was also a matter of compatibility for Moz. “We had met with them, we had talked with them, and we really liked the company,” Peters said.
Mighty AI created custom micro-tasks for its annotator community to gather the data Moz needed to train their algorithms. Community members were given an article or blog’s web page, and instructed to determine the author. “Through the platform, Mighty AI provided an easy and cost effective way to do this,” Peters shared. “We’re able to hand them a bunch of raw data and get back exactly what we need to develop our quality algorithms.”
To control—and later test—for quality, a Moz product manager completed tasks for 100-200 pages internally, giving some of the answers to Mighty AI to use as a benchmark.
When Moz received the first batch of data from Mighty AI, they compared the answers with what they’d done manually. “Where there were discrepancies, I looked, and in most of those cases—certainly more than half—I myself sided with the Mighty AI answer versus what the product manager had done. I was convinced the data was as good or better than what someone had done internally, and that to me was very convincing that I could trust the rest of the data,” said Peters.
“For us, the real value is that it’s very little effort on our end to collect the data and to send it to Mighty AI. Otherwise, [we’d be] essentially left with managing large projects and spending lots of internal time on Mechanical Turk or doing it manually,” Peters explained. “It’s the ability to have an easy way to collect data and it’s cost-effective. We were able to collect all the data we needed for this algorithm for little money considering the value it had for us.”
When asked if Moz would work with Mighty AI again, Peters said, “The team is very responsive about keeping us up-to-date on progress and how things are going on their end. When we have the need to collect data again, we will definitely contact them first.”
Will you be our next success story?
Let’s discuss how Mighty AI can help.