Blog on Big Data and Analytics

Big Data and Rise of Predictive Enterprise Solutions

February 10, 2012 by Harish Kotadia, Ph.D.

Given the three Vs of Big Data, namely Volume, Variety and Velocity (read this for more), challenge before large and medium sized companies is how to unlock the potential of Big Data and productively leverage its value in running the business.

In “traditional” Data Analytics or Business Intelligence, focus is more on analysis and reporting of “historic” or past data stored in the database. Take for example how most organizations use data from their CRM or ERP applications. Almost all the reports that are generated pertain to past or “historic” information. Running a business based on “historic” or past data is like driving a car looking in the rear view mirror and is not going to work.

Instead, companies must analyze all the available information in real time, apply statistical modeling techniques to available information in order to predict future outcomes and take action/run the business based on predicted outcome rather than analysis of historic data as is being done currently.

Since Big Data is characterized by not only Volume, but also Velocity and Variety, it is very important that Big Data is used for analysis in real-time to predict the future and take corrective action based on that analysis. How about using Causal Path analysis on Social Media data (like Twitter and Facebook) to predict Churn or Customer Attrition in Telecom industry and taking corrective action to prevent Churn/Attrition rather than analyzing “historic” attrition rate, call volumes or average response time as being done currently. The real value is in using predictive analytics and taking corrective action before it is too late, rather than just reporting historical information.

Techniques like Multiple-regression analysis coupled with Factor analysis, Cluster analysis and Causal Path analysis can be used very effectively with Big data – now that we have many variables and multiple observation for each variable at a customer level to generate statistically significant difference in analysis.

In future, no ERP or CRM system will be complete without Predictive Analytics functionality that will enable companies take preventive steps (rather than reactive) in real time. For example, rather than analyzing “historic” attrition rates, Predictive CRM application will make it possible for companies to identify critical incidents leading to customer attrition so that steps can be taken to retain the customer before he/she defects.

And thanks to Predictive Analytics, ERP or CRM applications will no longer be just a repository of “historical” information but will be transformed into a Predictive Knowledge base or Engine driving business decisions looking forward and not backwards/in the rear view mirror of “historic” information.

What do you think? Do you agree that Big Data will result in rise of Predictive Enterprise Solutions? Please do share your thoughts.

Big Data: Will Open Source Software Challenge BI & Analytics Software Vendors

February 3, 2012 by Harish Kotadia, Ph.D.

Predictive Analytics has been billed as the next big thing for almost fifteen years, but hasn’t gained mass acceptance so far the way ERP and CRM solutions have. One of the main reason for this is the high upfront investment required in Software, Hardware and Talent for implementing a Predictive Analytics solution.

As a result, only a handful of very large enterprises such as mega banks or top telecom companies have made the required investments and have benefited from power of Predictive Modeling and advanced Statistical techniques that are in existence for well over five decades. Most of the other companies have not been able to levarage power of business analytics as they cannot afford investing in specialized harware, database and BI/Analytics software applications being marketed by enterprise software vendors such as SAS and Teradata.

Well, this is about to change – thanks to technologies such as Apache Hadoop (which supports Big Data distributed applications under a free license), HBase (an open source, non-relational/distributed database) and the freely available R programming language (which is part of the GNU project). Using R, HBase and Hadoop, it is possible to build cost-effective and scalable Big Data Analytics solutions that match or even exceed the functionality offered by costly proprietary solutions from leading BI/Analytics software vendors at a fraction of the cost. And since R programming language is a freely available Open Source Software, users can leverage work done by others for specific analytics functionality and don’t have to re-invent wheel by rewriting the code. This reduces cost of developing analytics solution significantly.

Established BI and Analytics software vendors have no option other than offering their solution under SaaS (Software as a Service) model so that it is cost effective for their customers to implement analytics solution without requiring large upfront investment. This is all the more important for Big Data as the field is evolving rapidly. And if any BI or Analytics software vendor fails to adapt to this changing technological environment, they risk losing their market share.

Big Data Analytics a Key Enabler for Social CRM – Airlines Case Study

February 1, 2012 by Harish Kotadia, Ph.D.

Big Data Analytics is a hot topic of discussion these days. But many feel that it is more of a “hype” and less of substance. In my opinion, Big Data Analytics is the real deal and if used correctly, can deliver great business results at a fraction of cost compared to other alternatives.

Here’s a simple yet very effective example of using analytics for understanding consumer attitudes towards airlines in real time. In this study, Jeffrey Breen has used the R programming language to analyze consumer sentiments about major U.S. airlines expressed on Twitter. For more on the methodology and results, see the following embedded Slideshare presentation (if you cannot see the embedded file, click here to view it on Slideshare website).

As you can see on slide no. 27, the twitter sentiment scores obtained for many of the airlines are “comparable” to results of The American Customer Satisfaction Index (ACSI). What is important to note here is that by analyzing few tweets using freely available R programming language (which is part of the GNU project), it is possible to achieve results similar to that of an elaborate and expensive market research study such as ACSI at a fraction of a cost, that too in real time. Isn’t that a game changer?

Now imagine using Big Data Analytics to analyze Social Media data – not only text data but also pictures, video, GPS , Social Graph/Social Media linkages and using that information for engaging customers on Social Media channels. That in my opinion is the ‘holy grail’ (if I may use that term) of Social CRM.

In one of my earlier post, I have defined Social CRM as the business strategy of engaging customers through Social Media with goal of building trust and brand loyalty. Big data analytics is a key enabler for engaging customers on social media channels for building trust and loyalty (Social CRM). What do you think?

R by example: mining Twitter for consumer attitudes towards airlines

View more presentations from Jeffrey Breen

Why Big Data Analytics is The Next Big Opportunity for Outsourcing Companies

January 8, 2012 by Harish Kotadia, Ph.D.

Big Data Analytics is making big headlines these days. Just check out a few from recent past:

Big Data — Big Money Says It Is A Paradigm Buster - Forbes.com
For Start-Ups That Aim at Giants, Sorting the Data Cloud Is the Next Big Thing - The New York Times
Accel Makes Big Commitment To Big Data With $100M Fund - The Wall Street Journal
Five startup predictions for 2012 - c|net
Big Data: Investments Flow into the Next Big Thing - a great post by Dick Weisinger

So what is Big Data and why it is in the news so much these days?

According to Philip Russom, Director of TDWI Research, Big Data has three defining attributes – three Vs as he calls them. They are Data Volume, Data Variety and Data Velocity and together they constitute a comprehensive definition of Big Data. So Big Data is not just about Data Volume, but also the variety of data (mostly unstructured) and the velocity with which the data is generated and need to be analyzed. (for more, check out following posts by Philip Russom and the TDWI Best Practices Report titled Big Data Analytics):

Given three Vs of Big Data, ‘traditional’ data storage, retrieval and analytics methodologies are no longer going to work. Cloud Computing is going to play a key role when it comes to Big Data Management and Analytics. And here in lies the opportunity for Outsourcing companies.

Traditionally, data collected by organizations is ‘safely’ stored in massive relational database accessible to only few within the organization and requires elaborate infrastructure both in terms of hardware and software for storage, retrieval and reporting/analytics. In such an environment, it is not possible to easily outsource Data Analytics function/processes alone given the heavy investments made in terms of hardware and software.

Because of the business requirement of analyzing vast amount of ever changing structured and unstructured Big Data almost instantaneously, companies will be hard pressed to do this on their own. But given the fact that Big Data stored in cloud can be accessed from anywhere the internet is available and can be analysed almost instantaneously by third party service providers, outsourcing companies can offer to their clients value added services in the area of Big Data analytics without heavy investments on the part of clients in specialized hardware and software as was the case with ‘traditional’ data analytics. This will bring down significantly costs (especially fixed costs) associated with building and maintaining analytics infrastructure and solution center.

Just to give an example in the area of Social CRM, Social Media has empowered customers like never before as they can discuss about brands/products on Social Media channels. The best any marketer can do is to Listen, Learn and Engage customers. Given the three Vs of Big ‘Social’ data and the fact that most of the user generated content resides in the cloud, outsourcing companies can offer cost effective analytics solution to their clients to enable them effectively engage their customers/prospects in real time.

What do you think? Do you agree that Big Data Analytics is The Next Big Opportunity for Outsourcing Companies? Look forward to hearing your thoughts and comments:

3 New Year’s Resolutions that should be on Every CEOs List for 2012

December 30, 2011 by Harish Kotadia, Ph.D.

“After Outcry, Verizon Abandons $2 Fee” screams a headline from today’s online edition of the New York Times. And the article makes it very clear that Social Media played no small role in telecom giant’s hasty retreat from levying a new $2 fee on bill payments. This story is a harbinger of things to come in 2012 and CEO and CMO of every large and medium sized company should pay attention to it.

As I explained in my previous post titled 2012: Year of Empowered and Assertive Digital Customer, thanks to Mobile, Tablet computing and Social Networking revolutions, customers are empowered like never before and they are leveraging technology not only for shopping online, but also for highlighting poor customer service experiences. This growing trend of empowered and assertive customers leveraging technology for connecting with other like minded individuals to share their experience about products and services will be the big story of 2012.

In order to make sure that 2012 doesn’t become a year of Social Media disasters or Social CRM crises, CEOs and CMOs of large and medium sized companies should include following three new year resolutions on their 2012 list:

1. Improve collaboration within the enterprise and invest in collaboration tools

2. Listen to what customers/prospects are saying on Social Media channels and learn from it

3. Empower employees to engage customers/prospects on social media channels and invest in customer collaboration platform

How important are these to the survival and growth of business? In my opinion, these are as important as resolving to lose weight for someone who may suffer a heart attack if he/she doesn’t lose weight, or as important as bringing debt/expenses under control for someone who may face financial ruin or bankruptcy otherwise.

One may argue that these doesn’t apply to leading companies or brands that enjoy dominant position in their market. Well, think again. It doesn’t take long for assertive customers, empowered by social media to drive home their point of view. If you want proof, look no further than Netflix or GoDaddy in this regard. So as I said, better to lose weight than suffer a heart attack and better to control expenses/debt than face financial ruin.

Dear CEOs and CMOs, are you listening? Wouldn’t you include these three New Year’s resolutions to your 2012 list?

2012: Year of Empowered and Assertive Digital Customer

December 28, 2011 by Harish Kotadia, Ph.D.

As 2011 comes to an end and as we look forward to celebrating New Year 2012, one thing that strikes most looking back is the massive increase in ownership and usage of Smartphones and Tablet computers.

Just consider following headlines in the last few days:

“On Christmas day alone, a total of 6.8 million iOS and Android devices got activated around the world, a 353% increase from Christmas 2010″ (for more, see this link)

“Consumers have spent a record $35 billion online shopping this holiday season” (for more, see this link)

“Apple’s iOS accounted for a whopping 13.4% of Online Sales on Christmas Day” (for more, see this link)

“According to a recent report, Facebook reached more than half of global audience in October 2011 and accounted for 1 in every 7 minutes spent online around the world and 3 in every 4 social networking minutes” (for more, see this link)

These are just a few headlines I came across during last few day of 2011. And they all points to the fact that thanks to Mobile, Tablet computing and Social Networking revolutions, customers are empowered like never before and they are leveraging technology not only for shopping online or scoring great bargains, but also for highlighting poor customer service experiences.

This growing trend of empowered and assertive customers leveraging technology for connecting with other like minded individuals to share their purchase intentions and experiences about products and services will be the big story of 2012.

Questions is how many companies are listening to customers, learning from it and engaging them? Not many. So if 2012 is going to be the year of Empowered and Assertive Digital Customer and if companies are not ready for it, we can also say that 2012 is going to be the year of Social Media disasters and Social CRM crises, and many companies will learn the Social CRM lesson hard way. And that will spur investment not only in Social CRM, but also Enterprise Collaboration Platforms, Mobile engagement platforms and Data Analytics.

What do you think? Do you agree that 2012 is going to be the year of Social Media disasters and Social CRM crises? Please do share your thoughts:

(Picture courtesy: Beautiful sunset in Dallas, Texas as seen from my apartment)

FedEx learns Social CRM lesson, the hard way!

December 22, 2011 by Harish Kotadia, Ph.D.

Here’s an excellent example of the importance of responding to a Social Media crisis before it is too late. On December 19th, 2011, a YouTube user (username “goobie55″) uploaded video of a FedEx delivery person throwing a video monitor into his/her yard.

This YouTube video has created a Social Media storm of sorts and as I write this post, it has been viewed 3,576,412 times for this one YouTube post alone. I have seen it on my local TV news too here in Dallas, Texas.

Well, before it was too late (remember United Breaks Guitars YouTube video), FedEx posted its own response to customer’s video with the following statement by Matthew Thornton III, Senior Vice President, US Operations, FedEx Express:

“Along with many of you, we’ve seen the video showing one of our couriers carelessly and improperly delivering a package the other day. As the leader of our pickup and delivery operations across America, I want you to know that I was upset, embarrassed, and very sorry for our customer’s poor experience. This goes directly against everything we have always taught our people and expect of them. It was just very disappointing.”

(YouTube video response by FedEx embedded below):

This incident of Social Media brand crisis underscores the importance of having a Social Media Crisis Management Plan. It should clearly highlight roles and responsibilities, procedure and protocols to be followed in the event of a brand crisis. Because the first step in solving any crisis is to identify and respond to it, and respond FAST when it comes to Social Media as FedEx seems to have done here. What do you think?

Why ‘Social’ will drive growth in 2012

December 21, 2011 by Harish Kotadia, Ph.D.

Time flies! Here we are just a few days before Christmas and looking back at 2011, I cannot help but wonder at the amazing speed with which the year flew by! 2011 will be remembered for the effective role Social Media played in activist movements across the globe.

Looking at the future, one can see dark clouds on economic horizon given the debt and deficit crisis in Europe and North America. Consumer spending, which has driven economic growth for past several decades given easy availability of inexpensive credit, is drying up as consumers are reluctant to spend given the economic and political uncertainty. As a result, economic growth is almost 0% if not negative and with rising inflation, there is real risk of dreaded ‘stagflation‘ (economic stagnation with high inflation resulting in stubborn high unemployment).

In this scenario, companies are fighting hard for share of customers’ wallet/spend to meet their top line/revenue growth targets at the cost of bottom line/profits. As a proof of this, just note the amount of discounts large retailers and leading brands offered to customers this holiday season. And this is not just limited to budget brands for value conscious customers but also includes luxury brands targeted at affluent customers.

So how can companies not only survive, but thrive in an economic environment characterized by stagnation, high unemployment, eroding purchasing power because of inflation and when customers are unwilling to spend because of economic uncertainty? In economic parlance, this is a perfect storm and it requires some out of the box thinking on the part of companies.

Companies need to use emerging technological tools such as Social Media, Mobile and Cloud computing to drive down cost of doing business and at the same time, maximize value delivered to customers. Remember, it is not about price of a product or service but value perceived by the customer and price he/she is willing to pay for the perceived value of the product or service.

As I highlighted in one of my previous post titled Why Industrial Era Marketing Won’t Work in the Age of Social Media, not involving customers during product/service creation and delivery, and very limited after-sale support via phone is not going to work any more. Companies need to use social media channels and collaborate with their customers at every step of product or service creation and delivery.

To survive and grow in the age of social media, companies will have to re-engineer their product design, manufacturing, delivery and support processes, and not just their marketing processes, in order to involve customers at every stage of product/service creation and delivery. Companies that succeed in involving customers at every step of product/service creation and delivery will thrive and grow in this tough economic climate.

My suggestion to senior executives is that instead of viewing this tough economic climate as a challenge or a problem to overcome, why not view it as an opportunity for change by re-engineering business processes to take advantage of emerging technological tools such as social media and involve or engage customers at every stage of product or service creation and delivery. It can be as simple as a factory manager posting a YouTube video on how the product is made or a hotel employee tweeting about new mattress in all rooms in the hotel. By engaging customers on Social Media channels, companies can win greater mind-share of customers and as we all know, mind-share results in purchase and greater share of the wallet!

And before I end my post, I want to wish readers of this blog Merry Christmas and Happy Holidays.

(Picture courtesy: The well lit Christmas tree that you see above is from my apartment in Dallas, Texas)

Forget Black Friday, how about “Social Media Saturday”?

October 19, 2011 by Harish Kotadia, Ph.D.

Halloween is just a few days away and before we know, it will be Thanksgiving and more importantly for business, Black Friday and Cyber Monday (for those not familiar with the term Black Friday, see this).

But on the economic front, all seems to be ‘gloom and doom’ and ‘economic’ forecast for this holiday season seems to be dark, damp and cloudy with possibilities of storms – some severe. Question is can social media help marketers cheer up their holiday sales this season? or specifically, how can social media help boost sales on Thanksgiving weekend, which traditionally is the start of the Christmas shopping season.

Answer is yes, social media can be used very effectively for promoting sale of merchandise this holiday season. For example, how about announcing deals on company’s or store’s Facebook page rather than newspaper advertisements or promotional flyers? And to get the deal, customers will have to “Like” the deal. Better still, a customer can avail of the deal only if few of of his/her Facebook friends buy the product. More the number of Facebook friends who buy a product, higher the discount a customer get. Just imagine the excitement this will generate on Social Media and resultant free publicity.

And most online stores have option of Gift Shipping but what about having similar options on Facebook page of the company or the store, where a customer can select what items he/she want as Christmas Gift and this will be displayed on Customer’s Facebook profile letting friends and family know so that they can buy the desired gifts rather than something that is not going to be used or even worse, is going to be returned to store for refund/exchange – a big problem for retailers.

As I wrote in one of my previous post titled Social CRM: Thinking Outside the “Call Center” Box, we need fresh outside the box thinking to take advantage of unique characteristics of Social Media and Social Networks. Companies must leverage unique characteristics of Social Networks and re-engineer their business processes to derive full benefits from it.

If used properly Social Media can drive sales to such an extent that volume of sale on Social Media Saturday will be far greater than volume of sale on ‘traditional’ Black Friday with promotion only through flyers and advertisements.

What do you think? Do you agree that Social Media can help marketers cheer up their holiday sales this season? Please do share your thoughts. Would love to hear from you:

Why Industrial Era Marketing Won’t Work in the Age of Social Media

October 16, 2011 by Harish Kotadia, Ph.D.

In the industrial age, companies mass produced goods and promoted their products and services through mass media in order to sell them. Customers had little involvement in product creation (other than occasional feedback they provided through marketing research). And if customers had any problem, they contacted a support agent via mail or telephone.

Social media have radically changed this equation between companies and their customers by empowering customers like never before. Customers can discuss about products on social media channels and companies have no control over what customers are saying about their brands.

But instead of engaging customers on social media, most companies still rely on marketing the old fashioned way – a vestige from the industrial age! Marketing processes being followed today by most companies are still very similar to what was done in the industrial age.

Companies need to realize that when it comes to creation and delivery of products and services, expectation of “Social” customers are vastly different from that of customers in industrial age. “Social” customers expect companies to “listen” to them and take appropriate action on their comments regarding products and services they purchase and pay for.

Social media have provided tools for customers to express their opinion and customers are using Twitter and Facebook for the purpose. We see comments about brands/products on Twitter and Facebook all the time. For example, delayed flight or bad experience at a restaurant. But instead of listening to customers’ feedback on Social Media channels and engaging them, companies still expect them to call or write to them if they need any help. In a nut shell, companies are still living in the industrial age when customers have moved on into the social age.

In fact, I wrote about this almost two years’ back in a blog post titled “How Customer Engagement will determine winning brands in Social Era” In this post, I explained that level of customer engagement will determine mind-share and market-share for a brand. ONLY those companies that effectively engage their customers and prospects will emerge as winners and those that are not good in customer engagement will lose market share.

Unlike in industrial age, not involving customers during product/service creation and delivery, and very limited after-sale support via phone is not going to work any more. Companies need to use social media channels and collaborate with their customers at every step of product or service creation and delivery.

Companies that succeed in involving customers at every step of product/service creation and delivery will thrive and grow. Rest will become history. Question is how many senior executive realize the challenge that is in front of them and are prepared to make this transition?