Increasing prediction efficiency.
Mindhacks has an interesting take on the ongoing $1MM Netflix challenge to create an algorithm that will predict what unseen films customers will like based on their past preferences. But the bigger question is how can we reconcile numerical data with human thought and behaviour? Is human behaviour so predictable, that with available relevant data, we will be able to finally influence it to what is desired?
To predict preferences, what a company would typically do is look at subscriber past behaviour and form hypothesis. For example, families with kids at home might prefer more cartoons. But when these hypotheses are tested through experiments, all future predictions are more likely to be based on past conclusions, theories, even just hunches. Making logical decisions and hypotheses like these works at a superficial level.
Interestingly, what Netflix is doing is looking inside. It’s “If You Liked This, You’re Sure to Love That” algorithm identifies that common thread connecting movies across genres. It could be the extent of nerdiness in a movie, or maybe movies depicting certain nuances of urban life, or even movies with a particular type of indescribable humour. But now, as per the article, the SVD technique used in those self-learning algorithms has evolved to finding deep subconscious connections across movie genres that customers themselves wouldn’t even recognize.
Sure, our movie preference ranges across genres, but now the Netflix SVD algorithm can probably explain why Pretty Woman sits with the Terminator and Jerry Maguire in your list.
In these cases, it’s tempting to think there’s some deeply psychological property of the film that’s been captured by the analysis. Maybe all trigger a wistful nostalgia, or perhaps each represents the same unconscious fantasy…
This prediction is based on emotional decisions made by the subscriber. Its about decoding emotional data, not rational logical data. In cases like these, past behaviour can be a great indication of future behaviour, because emotions are empirical in nature. But remember, all the answers cannot come directly from the data. Interpretation of meaning is more important.
Experimental methods go from meaning to data, while exploratory methods go from data to meaning. Somewhere in the middle is our mind.