A couple of Wharton professors recently released a study of the distribution of demand for movie rentals at Netflix, based on the data the company released for the Netflix prize. The authors say the data contradict Chris Anderson’s long tail theory; Anderson says the data back up his theory; and Tom Slee says the data do neither.
I wonder, though, whether the Netflix data aren’t hopelessly skewed, at least when it comes to getting a sense of the relative demand for hits as opposed to less popular or niche titles. I’ve subscribed to Netflix for a long time, and what I’ve noticed is that the company has deliberately geared its search, filtering, and recommendation tools to lead customers away from newly released hits. There was a time, I’m pretty sure, when you could find a simple list of the week’s top new releases on the Netflix site. You can’t do that anymore. There is a New Releases tab on the Browse menu, but it brings you to an odd assortment of films that don’t bear much resemblance to the releases that are actually most in demand at the moment.
Here, for instance, is the first set of five movies that Netflix currently presents under the banner “Popular New Releases” (along with the actual DVD release date):
Obsessed (August 4)
The Soloist (August 4)
Confessions of a Shopaholic (June 23)
Revolutionary Road (June 2)
Seven Pounds (March 31)
Here, by contrast, is IMDB’s current list of the five most-rented DVDs in the country:
X-Men Wolverine
State of Play
Crank: High Voltage
Next Day Air
Duplicity
No overlap at all. You have to go down to #16 on the IMDB list before you find the first movie that’s on the Netflix list (Obsessed). In fact, you can scroll through Netflix’s “Popular New Releases” list all day long, and you will never come upon X-Men Wolverine or State of Play. And if you add the original X-Men movie to your queue, X-Men Wolverine will be conspicuously absent from the set of 10 movies that Netflix will immediately recommend as being “Most Like X-Men.”
By fussing around a bit, I was able to coax Netflix into giving me a list of “Movies Released in the Last Week.” Here are the first five I was shown:
Adam Resurrected
Lymelife
Road to Victory
Rage
The Anna Nicole Smith Story
And here are the next five I was shown:
Mr. Tickle: Tickle Time Around Town
Barney: Fun on Wheels
Scooby-Doo! The Mystery Begins
Mandie and the Secret Tunnel
Ghost Cat
Notable by their absence are the three most popular movies released on DVD this week: Observe and Report, Ghosts of Girlfriends Past, and Battle for Terra.
Now, to be fair, all the really popular DVDs can be found on the Netflix site, if, for example, you search by their name. But if you add any of them to your queue, you’ll be told that you’ll have either a short or a long wait until they ship. In the meantime, you’ll receive less popular titles from your queue. (You’ll be relieved to know, though, that the Mr. Tickle DVD is available immediately.)
By manipulating the movies it suggests, and by restricting the number of copies of new and popular movies it offers, Netflix shifts demand away from current hits and down the long tail. The reason, I think, is pretty obvious: the latest hits are the most expensive for Netflix to procure.* By manipulating demand, it makes more money (a venerable marketing strategy that’s given a new twist on the web). But it also spreads demand across its inventory in an artificial way that obscures its customers’ actual preferences.
In his Long Tail book, Chris Anderson talks about how the searching and filtering tools on the Net expose niche products that used to be difficult to find. That’s true. What I don’t recall him mentioning is that companies can use their search and filtering tools, as well as their inventories, to manipulate demand, deliberately leading customers, as in Netflix’s case, away from the hits and toward Mr. Tickle and The Anna Nicole Smith Story.** Sometimes we travel down the long tail under our own power. Sometimes we’re pushed.
*UPDATE: As noted in the comments, another and probably larger reason why Netflix tries to hide a popular new release is that it would have to buy a ton of copies of the DVD to fulfill the natural demand for the film, and after a few weeks, when the initial demand subsides, most of those copies would sit idly in its warehouses. By suppressing demand it avoids that expensive inventory overhang.
**UPDATE: My memory is flawed. As noted in the comments, Anderson does apparently mention demand-manipulation in the book.
It will be interesting to see whether Netflix’s strategy will change as it moves to streaming videos rather than to delivering DVDs.
Right now, Netflix doesn’t want to buy a million DVDs of this week’s blockbuster only to have them sitting and gathering dust in three months’ time, but digital delivery means that you never run out of stock so the incentive to drive people away from this week’s big hit is removed.
My guess is that Netflix’s strategy – and renters’ watching patterns – will depend on the details of the licensing agreements between Netflix and the movie suppliers, which is pretty much what you suggest.
It will be interesting to see whether Netflix’s strategy will change as it moves to streaming videos rather than to delivering DVDs.
That’s going to take quite a while.
But you raise an important point. The current Netflix streaming strategy also artificially promotes the long tail. Streaming is a free add-on to the regular subscription, and you can stream as many movies as you want. But the movies available for streaming are generally older and more obscure films (the ones, I suppose, that Netflix can stream for free or really cheaply). Assuming that people are as likely, more or less, to rate streamed movies as physical dvds, then film ratings (which is what the Wharton study uses as a proxy for rentals) would be further skewed toward the long tail – and perhaps strongly so.
Nick, the Long Tail book does make a passing reference to this demand shaping. It’s buried in the last few chapters of the book, not in the section about Netflix. In fact you and I exchanged emails about this very topic back in 2006.
Oliver – does he say anything worthwhile in his passing reference? I don’t recall anything beyond this frankly silly pair of paragraphs:
“What the VCR and the video rental store hinted at was the rise of the age of infinite choice. Those stores increased the available selection of movies of any given Saturday night a hundredfold.
Today, Netflix increases it a thousandfold. The Internet will increase it a gazillionfold. Every time a new technology enables more choice, whether it’s the VCR or the Internet, consumers clamor for it. Choice is simply what we want and, apparently, what we’ve always wanted.”
Thanks, Oliver, for correcting my memory. I’ve dug up your 2006 email on this subject, and here’s an excerpt: “…the Netflix data [in The Long Tail] has a serious flaw. While Anderson uses the company as an ‘excellent’ example of the long tail shift — comparing directly to Blockbuster sales — he later quietly reveals that Netflix under-buys hit DVDs thus constraining demand and forcing customers down the tail. While this may be good business practice, in my opinion it invalidates the comparison to Blockbuster and to some extent the example as a whole. We would need an example of a ceiling-free online retailer or Netflix DVD request data (not rental data) to properly understand what is going on in the market. It is my suspicion that if the Netflix demand was unaltered it would look much more like Blockbuster.”
My sense (as a subscriber) is that Netflix has since 2006 become more aggressive in shaping demand, particularly in the way it stages the browsing process and the way it offers recommendations.
Thanks Nick for pulling up the old email — i’ts funny, at the time I felt uncomfortable leaving a comment and figured email would be a better bet!
Looks like if it were available the DVD request data would also be a flawed comparison to Blockbuster, since the very site is geared towards pushing users down the tail, not just the inventory practices.
Tom, I’d have to dig up my copy of the book. The passage about DVD buying is in there somewhere.
Oliver – thanks, but no need, Nick’s comment makes it clear what you meant.
If 50% of a movie budget is for advertising, is that falsely creating demand at the head of the tail?
Lloyd, All marketing is an attempt to shape demand. The question is, do you accept it as such or do you pretend it is, or mistake it for, something else? Nick
Blockbuster was also constrained on the new releases. They seldom intentionally bought enough to satisfy the initial demand upon release of new material. They even laid them out along the perimeter of the store so you would walk all the way around, with other non top-10 titles mixed in.
Of course, they were trying to get you to rent more movies. Netflix makes money, in the short run, when you watch fewer movies. If you also end up with a “difficult” movie that you are seldom in the mood for, it clogs your buffer of DVDs to watch, until you chalk it up as a sunk cost and send it back unwatched.
I predict that a key thing we’ll learn from streaming is that people will bail out on a movie earlier, as the cost of switching to a different film is vastly reduced.
If you also end up with a “difficult” movie that you are seldom in the mood for, it clogs your buffer of DVDs to watch, until you chalk it up as a sunk cost and send it back unwatched.
So true. There have been many times when I’ve received an odd and unanticipated film from Netflix. I’ll say to myself, “Jeez, did I really put this in my queue?” And then it will sit by my TV for a number of days – each night I look at it and say, “nah I don’t think I’ll watch that now” – until finally I give up and send it back unwatched.
That makes 100,000 companies manipulating us to be more conformist and 1 company manipulating us to be less conformist. Go Netflix.
Seriously, this is one of the least evil instances of corporate manipulation that I’ve yet encountered.
Eliezer,
That’s a good and valuable point. And I certainly don’t recall that I or anyone else suggested that Netflix is “evil” for pursuing what I referred to as a venerable marketing strategy. The point of the post was to argue that Netflix’s manipulation of demand – whether it be good, bad, or neutral – rendered it unreliable as an example of the true pattern of consumer demand on the web.
But, anyway, it’s a good question: If a company, pursuing its own economic interests, manipulates its customers in a way that results in them being exposed to a greater diversity of cultural products, is that a good thing, a bad thing, or nothing?
(It should also be said that there’s a very practical benefit to customers from Netflix’s squeeze-the-head strategy: lower costs means lower subscription fees.)
Also, I’m curious as to whether there are other documented examples of companies using recommendation engines or other online filtering tools to manipulate customer choices to the companies’ benefit (rather than just direct customers to products they’re likely to be interested in).
Nick
I don’t know of such examples. They would be difficult to track down as recommender systems are proprietary black boxes.
Of course, there are examples of companies trying to improve the ratings they get on recommender systems, eg Belkin’s fake positive reviews on Amazon for its routers, and gaming Digg.
Nick,
I wanted to react to your latest statement:
>
I think that’s a very fine line to distinguish. It’s in any vendor’s interest to push high margin products as opposed to low margin ones. Chris Anderson’s Long Tail describes the retail mechanism as “attract customers with blockbusters and sell them high margin older products at the same time”. That’s manipulating customer choices too, in a way.
I don’t know how Amazon’s recommendations work, but I would be surprised if they didn’t skew the choices towards higher margin products first.
Last year I got dragged into a somewhat unhealthy (from my point of view) debate about the Long Tail (see http://www.fiberevolution.com/2008/11/cutting-the-long-tail-short.html and http://www.fiberevolution.com/2008/11/long-tail-dressing-down.html) when my main contention point was that the data set (suspected to be iTunes UK) was too far from an unskewed market to draw any conclusions as to the legitimacy of the Long Tail “theory”.
I now think that there is no such thing as an unskewed market. What Netflix demonstrates (to me) is that you can exploit a Long Tail market profitably. That doesn’t mean that, as a market, it would naturally gravitate towards Long Tail purchasing characteristics.
In fact, the search engine / filter argument weighs so much into Chris’ thinking as to be an integral part of the model. Inevitably, “efficient” search and filtering is going to be skewed one way or another. We may all have accepted that Google’s algorithm in search is neutral, but even if there’s no deliberate tampering, the algorithm itself is non-neutral. It assumes that the most viewed / linked information is the most relevant.
Here’s a workaround:
RottenTomatoes lists weekly top rentals at http://www.rottentomatoes.com/dvd/top_rentals.php
This site integrates with Netflix — you can add movies to your queue from the RottenTomatoes site.
As a bonus, then go to http://www.redbox.com to reserve a movie for a buck… there are 4 Redboxes within 1/2 mile of where I live.
As to the very term “skew”, it strikes me that there is no natural state of “unskewedness”. Advertising might drive demand, but it also simply provides basic awareness and information.
Would “unskewed” mean NO advertising? That makes no sense…no one would know about products in order to be able to buy them.
Definitely looks like they are running a pre-filtering script to selectively remove certain strings/titles BEFORE it is passed to the search engine. Their engine appears to be doing a keyword search inside the titles rather than whole title searches without intelligent disambiguation which might explain some of the skewed results. I think I remember reading in one of my statistic classes that the standard normal curse only applies to large( but not huge) population samples, and that theoretically if the sample get huge (billions?) that the tails of the curve begin to disappear and the whole curve flattens out making any assumption about a “tail” irrelevant.