The Netflix Prize

Netflix announced its $1 million challenge in October 2006.  And the contest has now been won.  The big news is that collaboration was the key.

The Netflix web site has an automatic program that recommends movies that it thinks you will like, based on the number of stars you give to movies you’ve already rented.  They call this automatic program Cinematch.  Netflix can measure how well Cinematch works, by comparing the predictions it makes about how much you’ll like a movie, with the number of stars you actually give it after you watch it.  They have a bunch of very smart computer scientists who developed Cinematch, and in 2006 it was already very good.  So the Netflix challenge was going to be hard to beat: If you can develop your own prediction program and you can make it 10 percent better than Cinematch, you get one million dollars.

On Sunday July 26, two different teams both announced they had crossed the 10 percent better threshold.  Netflix is now comparing the two and will announce the winner in September.

The biggest lesson learned, according to members of the two top teams, was the power of collaboration. It was not a single insight, algorithm, or concept that allowed both teams to surpass the goal…Instead, they say, the formula for success was to bring together people with complementary skills and combine different methods of problem solving.*

The first team to win, BellKor, was a seven-member group.  None of the runners-up were solitary individuals; every solid contender, it turns out, was a team working collaboratively.  Contest rules then kicked in a 30-day period for any other entrants to give it their last best shot. What happened was that all of the leading teams merged together to combine their best ideas; a global team of about 30 members raced to beat the 30-day clock.  They called themselves The Ensemble.  As of July 26, The Ensemble seemed to be marginally better at predicting than BellKor: one-hundredth of a percentage point.  But those measures are submitted by the teams, and Netflix itself is going to take a couple of months making the official calculations…so, no winner will be announced until September.

David Weiss, a doctoral student in computer science at the University of Pennsylvania, and a member of The Ensemble, concluded:

The contest was almost a race to agglomerate as many teams as possible. The surprise was that the collaborative approach works so well.*

*Steve Lohr, “Learning the Power of Teamwork In a Netflix Race for $1 Million,” New York Times, Tuesday, July 28, 2009, pp. B1, B7.

5 thoughts on “The Netflix Prize

  1. I hadn’t read about this, so thanks for the story. I’m especially intrigued that the top teams “raced” to share ideas and collaborate. At some point they had to put aside their idea of winning as a single, smaller numbered team and recognize that the only way to accomplish their goal was to collaborate even more than they had been. Also somewhat surprised that David Weiss was “surprised that the collaborative approach works so well.” Wonder exactly what he meant – worked so well in terms of a solution (that doesn’t seem surprising to me anyway) or worked so well as a process and set of interactions.

  2. Good point…I’m not sure exactly what he was surprised about. I assumed he had been expecting solitary efforts to be just as, or more, effective.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s