the market conversation

In every market there is a conversation. That conversation was offline until recently. Now a lot of it is online, and that means we can listen in. When we can listen in we can learn things earlier, and when we can do that we can let the market know what its talking about in a way that enables the conversation to be broader, faster, more informed and more prolific. And then we can listen to that.

In business to business markets the publisher/media brand's role and purpose is to be a host to, and report on, the conversation of the market. Now that the conversation is moving, will that change?

Collective Intelligence

I was at the CEOTech conference this week at Stanford put on by Chief Executive. Good agenda, good speakers - David Kim is an interesting guy for sure.

One of the sessions was about "Cloud Computing + Crowd Sourcing + Collective Intelligence = Higher Success", and was moderated by successful angel investor M.R. Rangaswami. OK, the cloud computing piece was fairly easy to discuss (Amit Singh from Google made the best analogy when he compared cloud computing to electricity - now we only think about the electric outlet not the task of generating power in our business).

However, the panel really fell apart in my view when it came to explaining and differentiating crowd-sourcing and collective intelligence. I'm sure the crowd was collectively confused by the end. I'm not blaming the panel but they really described crowd-sourcing as a low-cost, high reach RFP system - find a designer on Elance or a developer on Odesk type of thing. Then they moved on to "collective intelligence" and really described it - to all intents and purposes - as 'Ask the Audience' on Who Wants to Be a Millionaire.

Collective intelligence is the aggregated or average views or actions taken from a large number of people (actively or passively) for the purposes of enabling decisions and insight into a topic or fulfilling a task. Crowd-sourcing is a tool you can use to do this. Survey and research companies have been doing this for years, the point now is that tools and availability means we can all do it and it has a potentially big impact on our business.

Humor Tracker

365 Media's R&D team are working away on our revolutionary new Social Media content tracker - we're weeks from launch. In the course of our work we look at a lot of 'anomalies' - trying to determine what is being said. Some we just have to highlight..

In Summary

To build an effective content summarizer is in many ways the same task as building an effective topical analysis and filtering system for unstructured content.

An effective summarizer - a program that 'reads' a lengthy article and then provides a natural language (or even bullet points for now, we can handle that) summary of the 'pertinent' points of the article means determining the key topics and then extracting and re-phrasing those points in shortened format. So, the challenge is not only what are the topics, but what are the "key" topics.

Then we get into subjectivity. The ultimate arbiter of what are the key topics is the author, but different readers may get different values from the content so ultimately the decision on what the key topics are depends on the needs of the audience. This reader is looking for information on nanotechnology, is that a key topic in the piece or just a passing mention?

These are the challenges. What's the key topic of this blog entry? Well, I'm not going to make it easy and put any labels on it..

Positive Reinforcement

I spoke at the NFAIS 11 annual conference last week in Philadelphia. As I was speaking, my colleagues back in the office were tracking tweets from the conference audience describing what I was saying. They wanted to run a sentiment analysis on the tweets and then text me an update during my talk as to how I was doing. Unfortunately the tweets were non-sentimental, just reportage. Wouldn't it be great having a real-time sentiment analysis feedback system? I know in the last presidential election they had the audience pressing buttons up and down to express their sentiment as the presidential candidates squared off on a live debate, but I'm talking about bigger than a few hundred and unstructured. Only a matter of time..

Anyway, I'll take it as good thing no-one expressed negative sentiment about my talk. John Blossom wrote a very good summary - way better than anything any existent automated content summarization tool we know about or have made ourselves could do - here.

Content Quality

A while back, when 365 Media was figuring out how to determine which information sources were more reliable than others - we developed the concept of the 'content momentum' of a site. Our NLP-based system were scouring content to find described business events to convert into data changes in existent (customer) databases. We were constantly looking for ways for our system to make better decisions based on things like priority (which news piece is more reliable), efficiency (which site is churning out re-runs and wasting good server time) and ranking (in this industry, which are the best sources of primary news).

Content Momentum was a concept that enabled us to do this - it worked loosely around the mass (size of overall content on site) multiplied by velocity (rate at which new content appears on the site) concept (hence 'momentum') with added caveats around the value of that content as it was processed through our system and then fed back to the training set.

All very interesting. Why am I mentioning this? Well, today Google announced a change to its search algorithm in order to "provide better rankings for high-quality sites—sites with original content". Google now cares whether the site has content quality? Well, duh, that's great.

Process This

Mainstream media has temporarily got caught up in the discussion about meaning-based computing and processing with IBM's Watson kicking some champion butt on Jeopardy! My first thought when I heard about this a few month's back was - why is this IBM? OK, the Deep Blue chess master and all that, but this is the leading edge of software. Why is this not Microsoft, Autonomy or another software company leading the way in showing how NLP has broader application?

Then I went to the IBM website and I realized the answer. Watson is a marketing gimmick to demonstrate the power of IBM's Power7 processor. IBM's marketing team are using natural language processing as a theme to show how powerful it's hardware is, because natural language processing requires a heck of a lot of, well, processing. So, in some ways this both undermines and supports the sense that NLP has real world application by firstly showing it in a mainstream application (answering questions on Jeopardy!) but secondly where "a single, precise answer to a question requires custom algorithms, terabytes of storage and thousands of POWER7 computing cores working in a massively parallel system".

OK, so bringing in Moore's Law, that's how long til I can give my laptop a name and it can make me some money on Millionaire?