Text Analytics and Predictive Modeling

Seven weeks as Director of Advanced Analytics at Attensity have raced by and I have been in total immersion mode understanding the Attensity software from front to back, understanding customer use of Attensity software, and building requirements for advanced analytics.  Customers are already integrating Attensity results with other mainstream analytics software.  Some of these customers also have competing text analysis software.  But the competitor software, in the use cases I have seen to date, has an entirely different approach to analyzing a set of documents.

Approaches to Natural Language Processing

The typical approaches most text analytics vendors employ for natural language processing (NLP) solutions include shallow parsing. The Attensity deep parsing provides, as its name suggests, a deep or detailed linguistic analysis of documents (think high school English and sentence diagramming). This means that a more detailed understanding of context is possible – and there’s no comparison when it comes to using deep parsing for sentiment analysis of customer or social media data.  Shallow parsing provides a partial understanding of a document set.  A detailed understanding of the context within a document is not the goal of a shallow parser.

Diving into deep parsing

The automatic Exhaustive Extraction technology used by Attensity is based on deep parsing. The underlying technology helps you understand questions such as “who did what to whom?”  Consider the following sentence: “The bolt cracked on the transmission case and resulted in a fire”.  Deep parsing helps you understand that the root cause of the problem was the cracked nut.   Shallow parsing would know there was an incident with a bolt, the transmission case and a fire but not the root cause.  Another example: “The motor vehicle driver ignored the red light at the intersection and collided with the motorcycle”. What is the root cause of this problem?  The car driver who ran a red light.  Important liability information for an insurance company! Shallow parsing would know there was an incident with a motor vehicle and a motorcycle but not who was at fault.

Business use cases for modeling

The detailed information automatically extracted from documents can be used for descriptive modeling and as input to predictive models. Descriptive modeling (or unsupervised learning) attempts to describe customer, transaction and social data using techniques such as segmentation and cluster analysis. Predictive modeling (or supervised learning) attempts to build a model to predict a certain outcome such as churn, satisfaction, fraud, foreclosure, response to a solicitation and much, much more.

Predictive Modeling

Consider a model that predicts whether you should be granted a loan or not.  Lending institutions build predictive models using information about current customers where the outcome (whether to grant a loan or not) is already known.  These models are then used for new customers or new loan applications where the current outcome is unknown.  When you apply for a loan at a bank, you supply information about your income,   expenses, number of years in your current job, etc. The bank uses a predictive model to input all of the known information to come with an answer to the unknown information – whether to grant you a loan or not.   The same thing goes for satisfaction. Using information like demographics and purchasing behavior about customers who have already indicated their level of satisfaction a model can be built to predict satisfaction of a new customer where satisfaction is unknown or maybe cannot be gathered directly from the customer. The prediction of loan approval (or not) or satisfaction (or not) itself is a useful measure as is the explanation for WHY the model approved the loan or predicted satisfaction. Predicting that a vehicle might be involved in a crash is one thing, explaining why (sudden acceleration, tread separation, brake system failure) is also very useful.   The prevalence of consumer opinion either on customer databases or on internet chat rooms, blogs, social media sites, means that there is the potential to improve your model accuracy by including information coming from unstructured data sources such as text or voice.

Customers are already seeing examples of model improvement, or lift, due to the addition of this valuable metadata derived from the text. Plentiful product reviews and ratings such as those available on Amazon.com and social media sites mean that customer commentary can easily be used to build predictive model s. The information extracted using our deep NLP engine also allows for better explanation of data mining models. I know I have said it before, but I think it is worth repeating: customers find amazing ways to innovate with text!

In the coming weeks and months I hope to post some specific use cases of customers who are using Attensity output for advanced analytics applications.  If you already have models built or in production or if you are experimenting, I would love to hear from you!

Photo credits: bpende, markchapmanphoto, peasap, kevinzhengli