A Certain Sentiment in the Air
by Jeff Fried
Can machines understand the feelings expressed in text? Social media (including blogs, discussion groups, private emails, etc) and mainstream press (web pages, newswires, announcements, etc) are full of quirky, subtle cues about people’s attitudes and opinions. Harvesting this information has proven to be valuable for tracking companies’ reputations, playing financial markets, analyzing customer loyalty, tracking brand effectiveness – and lots of other applications. But doing this by hand is so expensive and inconsistent that only a few market research shops do it.
Machine “understanding” of sentiment – which promises to unlock the information in social media in particular - is a technology that has been part of search as well as a field called text analytics for years. We may have reached a “tipping point” for this technology – at least that’s my impression from my recent visit to the 4th annual Text Analytics Summit last week. 
Text Analytics is one of the “adjacent” areas that overlaps search, and the boundaries are getting fuzzier by the year. It remains a small specialty market but it has grown well beyond its roots with government and pharma customers, and this year’s summit had a nice buzz to it (in contrast to last year, which was noticeably flat). Given the blurry adjacencies to Enterprise Search and Business Intelligence, and the rash of acquisitions over the last year, one can’t help wonder how long it will be until text analytics is no longer its own market. At the vendor and analyst panels, many of the panelists thought text analytics would become subsumed into applications (like Voice-of-the-Customer or market-influence-analysis), acquired into search and BI, or both.
One of the interesting things to me was that sentiment analysis was everywhere. The vendors exhibiting all highlighted sentiment analysis, the Microsoft Research talk demo’d sentiment analysis, half of the talks mentioned sentiment analysis. Nick Patience of the 451 group listed 20 vendors supplying sentiment analysis, and he missed a few I know of. Andrew Bernstein of Cymphony (a longstanding provider of “market influence analytics”) went so far as to say that an analysis of social media was practically useless without it.
But as far as I can tell the technology hasn’t changed dramatically over the last couple of years. FAST customers have been using sentiment analysis for about 4 years now, and many of the offline text-blob analysis applications (which FAST doesn’t do but which a number of vendors do quite well) have been around for equally as long. So what’s changed? Why the extra buzz this year?
Some people seemed to think this was the ebb and flow of the market; other folks felt the tipping point had been reached, or that acquisitions of ClearForest, Teragram, Inxight, etc had raised the level of interest and broadened the sales channel.
I think there’s another factor, too – business analysts are getting used to fuzziness, and the packaging of this technology has become simple enough to reach the point that you don’t need to be a linguist to use it. This technology will never be perfect, but it is good enough to add real value, and I think it gets even more traction when you stop focusing on how it works and start just using it. I heard none of the usual disclaimers about how machines can’t understand sarcasm, which was refreshing – this group at least knows that there are always limitations in the machine understanding of human language, and seems to have moved on.
Whatever the reason, it was fun to see some real success stories – real companies getting real benefits out of analyzing their unstructured text. Of course, it’s always fun to party with other people who care about the surface forms of sentiment, and like finding new ways to combine statistics and linguistics. But the feeling at this summit was different than I expected – more real, more pragmatic, more mainstream. And there was a certain sentiment in the air…
FAST
Learn how Reed Business, a member of the Reed Elsevier Group and largest B2B publisher in the world, is winning the premium content war by transforming their business from content creator to a leading vertical search provider. Graeme McCracken and Kate Worlock will discuss the approach Reed has taken to strategically leverage search and semantics across its portfolio of properties to provide a superior information discovery and consumption experience for its community and a flexible foundation for Reed Business.
