The Netflix Prize August 12, 2009Posted by keithsawyer in Genius Groups.
Tags: bellkor, cinematch, netflix, netflix prize, recommendation engine, the ensemble
Netflix announced its $1 million challenge in October 2006. And the contest has now been won. The big news is that collaboration was the key.
The Netflix web site has an automatic program that recommends movies that it thinks you will like, based on the number of stars you give to movies you’ve already rented. They call this automatic program Cinematch. Netflix can measure how well Cinematch works, by comparing the predictions it makes about how much you’ll like a movie, with the number of stars you actually give it after you watch it. They have a bunch of very smart computer scientists who developed Cinematch, and in 2006 it was already very good. So the Netflix challenge was going to be hard to beat: If you can develop your own prediction program and you can make it 10 percent better than Cinematch, you get one million dollars.
On Sunday July 26, two different teams both announced they had crossed the 10 percent better threshold. Netflix is now comparing the two and will announce the winner in September.
The biggest lesson learned, according to members of the two top teams, was the power of collaboration. It was not a single insight, algorithm, or concept that allowed both teams to surpass the goal…Instead, they say, the formula for success was to bring together people with complementary skills and combine different methods of problem solving.*
The first team to win, BellKor, was a seven-member group. None of the runners-up were solitary individuals; every solid contender, it turns out, was a team working collaboratively. Contest rules then kicked in a 30-day period for any other entrants to give it their last best shot. What happened was that all of the leading teams merged together to combine their best ideas; a global team of about 30 members raced to beat the 30-day clock. They called themselves The Ensemble. As of July 26, The Ensemble seemed to be marginally better at predicting than BellKor: one-hundredth of a percentage point. But those measures are submitted by the teams, and Netflix itself is going to take a couple of months making the official calculations…so, no winner will be announced until September.
David Weiss, a doctoral student in computer science at the University of Pennsylvania, and a member of The Ensemble, concluded:
The contest was almost a race to agglomerate as many teams as possible. The surprise was that the collaborative approach works so well.*
*Steve Lohr, “Learning the Power of Teamwork In a Netflix Race for $1 Million,” New York Times, Tuesday, July 28, 2009, pp. B1, B7.