Big Data: Are You Ready for Multimedia Information Systems?

If you take a look at Enterprise Information Systems that are currently in use, you will notice that they are very restrictive in nature in the sense that it is quite difficult to handle data types other than text or numbers. Moreover, text and numbers are handled using pre-defined (text and numbers) fields only in the database. It is extremely difficult to store/retrieve files of other data type like image or sound. This can be attributed to the fact that although we have seen remarkable changes in computing power, storage capacity and network speed/reach, the underlying database technology hasn’t changed much in the past two decades.

Current generation of Enterprise Information Systems use Relational Database Management Systems (or RDBMS for short). RDBMS is a Database Management System in which not only data is stored in tables, but relationships among the data are also stored in tables. RDBMS use Structured Query Language (or SQL) for managing data and SQL places data in well defined structures using meta data.

Given three Vs of Big Data, namely Data Volume, Data Variety and Data Velocity, traditional data storage, retrieval and analysis tools/technology like RDBMS and SQL  are no longer going to work and we are going to leap frog into a new generation of technology that will fundamentally change the way data is stored, retrieved and analyzed.

Big Data revolution has resulted in the development of a whole host of new tools and technology that can meet the challenge posed by three Vs of Big Data. For example, Not Only SQL (or NoSQL for short) doesn’t adhere to RDBMS model and is designed to capture all data without categorizing and parsing upon entry into the system. Using NoSQL and other tools such as HBase, Hive etc., it is possible to store, retrieve, and analyze very large data sets of multiple data types not just text and numbers. As a result, we are about to enter into an era of multimedia enterprise information systems not only capable of handling very large data sets in real time but also capable of handling multiple data types like sound, video etc.

Question is how many large and medium enterprises are ready for Multimedia Information (ERP or CRM) Systems? Not many in my opinion. What do you think?

4 Excellent Big Data Case Studies

In response to my previous post titled Big Data: The Coming Sensor Data Driven Productivity Revolution, some one asked me if I had few good case studies that highlight effective use of Big Data. I would like to share following four excellent Big Data case studies that shows how large corporations have started leveraging Big Data for driving productivity and increasing ROI:

TXU Energy – Smart Electric Meters:

In my blog quoted above, I indicated that “Because of smart meters, electricity providers can read the meter once every 15 minutes rather than once a month. This not only eliminates the need to send some one for meter reading, but as the meter is read once every fifteen minutes, electricity can be priced differently for peak and off-peak hours. Pricing can be used to shape the demand curve during peak hours, eliminating the need for creating additional generating capacity just to meet peak demand, saving electricity providers millions of dollars worth of investment in generating capacity and plant maintenance costs.”

Well, I have a smart electric meter in my home and one of the electricity providers in my area (TXU Energy) is using the smart meter technology to shape the demand curve by offering “Free Night time Energy Charges — All Night. Every Night. All Year Long.” (See this for more).

In fact, they promote their service as “Do your laundry or run the dishwasher at night, and pay nothing for your Energy Charges”. What TXU Energy is trying to do here is to re-shape energy demand using pricing so as to manage peak-time demand resulting in savings for both, TXU and customers. This wouldn’t have been possible without Smart Electric meters.

T-Mobile USA has integrated Big Data across multiple IT systems to combine customer transaction and interactions data in order to better predict customer defections. By leveraging social media data (Big Data) along with transaction data from CRM and Billing systems, T-Mobile USA has been able to “cut customer defections in half in a single quarter”.

US Xpress, provider of a wide variety of transportation solutions collects about a thousand data elements ranging from fuel usage to tire condition to truck engine operations to GPS information, and uses this data for optimal fleet management and to drive productivity saving millions of dollars in operating costs.

McLaren’s Formula One racing team uses real-time car sensor data during car races, identifies issues with its racing cars using predictive analytics and takes corrective actions pro-actively before it’s too late! (for more on T-Mobile USA, US Xpress and McLaren’s F1 case studies refer to this article on FT.com)

Big Data: The Coming Sensor Data Driven Productivity Revolution

When it comes to Big Data, we often think of user (human) generated social media data. But sensor (machine) generated data is a much bigger story as sensor data will drive the next wave of productivity growth and innovation. Consider this for example, “Boeing jet engines can produce 10 terabytes of operational information for every 30 minutes they turn. A four engine jumbo jet can create 640 terabytes of data on just one Atlantic crossing, multiply that by the more than 25,000 flights flown each day” (Source: Information Management). Almost all of this data is lost on completion of the flight, but this is about to change.

Thanks to Big Data tools and technology, we can identify, store, retrieve and analyze this data in a cost effective and timely manner. Think about the possibilities this opens up in the area of preventive maintenance and fault prevention – resulting in reduced flight delays and cancellations because of technical issues with the plane. And given the cost effectiveness of Big Data tools and technology, vast amount of sensor data can be analyzed in almost real time in a wide array of fields, not just in aircraft maintenance.

Smart electric meters are another good example of sensors generating vast amount of data that can be used to drive productivity. Because of smart meters, electricity providers can read the meter once every 15 minutes rather than once a month. This not only eliminates the need to send some one for meter reading, but as the meter is read once every fifteen minutes, electricity can be priced differently for peak and off-peak hours. Pricing can be used to shape the demand curve during peak hours, eliminating the need for creating additional generating capacity just to meet peak demand, saving electricity providers millions of dollars worth of investment in generating capacity and plant maintenance costs.

These are just two examples. Now imagine the possibilities of what can be done with a vast array of sensor data that is analyzed and used in industrial/business processes in real-time. In my opinion, we are on the cusp of a major sensor data driven productivity revolution that will fundamentally change the way we do business, for the better!

 

 

 

 

Missing Wood for Trees, Why Big Data Isn’t About the Big Data

Big Data is a hot topic of discussion these days and is on top of “To Do” list of senior executives of large and medium sized companies, many of whom have started investing in required infrastructure for leveraging Big Data.

While this is a good start, it is important to remember that Big Data isn’t about the Big Data. Yes, you read that correctly – Big Data isn’t about the Big Data, rather it is all about making better business decisions using the insights gained from analyzing structured and unstructured data that is available. If one focuses solely on “Big Data” and forgets the business context/how the data is going to be used, one is likely to miss wood for trees.

Instead of focusing on Big Data per se, how about starting with business problems that we are trying to solve, for example reducing customer defection/churn or improving cross-sell/upsell rates in CRM context or reducing inventory carrying costs in ERP context.

Once you have identified the problem to be solved, identify what structured and unstructured data are required for making better business decisions. This may include Social Media data for example besides ‘internal’ CRM data. Once you have identified the structured and unstructured data, internal and external to the organization, formulate a strategy on how the data is going to be used or analyzed and at what level of granularity. Only after this should one decide on what tools to use and make required investments in Big Data infrastructure and not other way around.

So don’t invest in Big Data infrastructure first not knowing what, why and how, and put cart before the horse. Rather identify a business problem to solve first, identify what structured and unstructured data are required for making better business decisions, develop a use case for Big Data, run a  successful pilot project and then invest in required Big Data infrastructure.

What do you think? Do you agree that Big Data Isn’t About the Big Data, rather it is all about making better business decisions using the insights gained from analyzing structured and unstructured data? Look forward to hearing your thoughts and comments:

 

 

Big Data: $16.9 Billion opportunity for IT Services and Consulting Industry

International Data Corporation recently released its “Big Data technology and services forecast” predicting that Big Data market will grow from $3.2 billion in 2010 to $16.9 billion in 2015, a compounded annual growth rate of 40% and “growth of individual segments from 27.3% for servers and 34.2% for software to 61.4% for storage”. So what is driving this growth in Big Data and how can IT Services and Consulting companies tap into this phenomenal growth opportunity of 40% CAGR, which according to IDC is “about 7 times that of the overall information and communications technology (ICT) market”?

As I indicated in one of my previous post titled Why Big Data Mining/Analytics is the New Gold RushBig Data is a key enabler for Social Business and without Big Data Mining/Analytics, a large or medium sized company can neither make sense of all the user generated content online nor can collaborate with  customers, suppliers and partners effectively on Social Media channels. And both these activities, namely gaining insights from user generated content online and collaborating with customers and partners are critical for success in the age of Social Media. Big Data provides the critical backbone for Social Business leveraging advances in Cloud Computing, Mobile Platforms, Social Media, Ubiquitous broadband internet, Gaming technology and Data Storage and Analysis.

Because of the business requirement of analyzing vast amount of ever changing structured and unstructured Big Data almost instantaneously, companies will be hard pressed to do this on their own. But given the fact that Big Data stored in cloud can be accessed from anywhere the internet is available and can be analysed almost instantaneously by third party service providers,  outsourcing companies can offer to their clients value added services in the area of Big Data analytics without heavy investments on the part of clients in specialized hardware and software as was the case with ‘traditional’ data analytics. This will bring down significantly costs (especially fixed costs) associated with building and maintaining analytics infrastructure and solution center. I expect all major IT Services and Consulting companies to invest heavily in building delivery capability in the area of Big Data/Analytics to tap into this opportunity.

Do you agree that Big Data is a great opportunity for IT Services and Consulting companies? Please do share your thoughts:

Why Big Data Mining / Analytics is the New Gold Rush

Just mention the words “Big Data” to any technology entrepreneur or investor and observe how his/her face lights up with excitement. Given the perceived opportunity in Big Data, tech entrepreneurs and investors want to capitalize on it by starting /investing in a Big Data Management, Mining and Analytics business. Is this perceived opportunity in Big Data for real or is it a bubble that will burst soon?

I think the perceived opportunity in Big Data is real and is here to stay as Big Data Mining/Analytics will fundamentally change the way business is done not only online but also offline. Here’s why: Big Data is a key enabler for Social Business and without Big Data Mining/Analytics, a large or medium sized company can neither make sense of all the user generated content online nor can collaborate with  customers, suppliers and partners effectively on Social Media channels. And both of these activities, namely gaining insights from user generated content online and collaborating with customers and partners on Social Media channels are critical for success in the age of Social Media.

Can you imagine any large or medium sized business without e-Business in the internet age? Similarly, it is impossible to run a large or medium sized business in the age of Social Media without leveraging user generated content or without collaborating with customers and partners. And for this to happen, Big Data Management, Mining and Analytics are critical/key enablers.

Being a key enabler and catalyst in Social Age, Big Data Management, Mining and Analytics companies are going to command a premium valuation and hence Big Data Mining and Analytics has triggered a new ‘Gold Rush‘, not to mine gold this time but something even more valuable – knowledge and insights.

What do you think? Do you agree that Big Data has triggered a new Gold Rush?

 

Big Data and Rise of Predictive Enterprise Solutions

Given the three Vs of Big Data, namely Volume, Variety and Velocity (read this for more), challenge before large and medium sized companies is how to unlock the potential of Big Data and productively leverage its value in running the business.

In “traditional” Data Analytics or Business Intelligence, focus is more on analysis and reporting of “historic” or past data stored in the database. Take for example how most organizations use data from their CRM or ERP applications. Almost all the reports that are generated pertain to past or “historic” information. Running a business based on “historic” or past data is like driving a car looking in the rear view mirror and is not going to work.

Instead, companies must analyze all the available information in real time, apply statistical modeling techniques to available information in order to predict future outcomes and take action/run the business based on predicted outcome rather than analysis of historic data as is being done currently.

Since Big Data is characterized by not only Volume, but also Velocity and Variety, it is very important that Big Data is used for analysis in real-time to predict the future and take corrective action based on that analysis. How about using Causal Path analysis on Social Media data (like Twitter and Facebook) to predict Churn or Customer Attrition in Telecom industry and taking corrective action to prevent Churn/Attrition rather than analyzing “historic” attrition rate, call volumes or average response time as being done currently. The real value is in using predictive analytics and taking corrective action before it is too late, rather than just reporting historical information.

Techniques like Multiple-regression analysis coupled with Factor analysis, Cluster analysis and Causal Path analysis can be used very effectively with Big data – now that we have many variables and multiple observation for each variable at a customer level to generate statistically significant difference in analysis.

In future, no ERP or CRM system will be complete without Predictive Analytics functionality that will enable companies take preventive steps (rather than reactive) in real time. For example, rather than analyzing “historic” attrition rates, Predictive CRM application will make it possible for companies to identify critical incidents leading to customer attrition so that steps can be taken to retain the customer before he/she defects.

And thanks to Predictive Analytics, ERP or CRM applications will no longer be just a repository of “historical” information but will be transformed into a Predictive Knowledge base or Engine driving business decisions looking forward and not backwards/in the rear view mirror of “historic” information.

What do you think? Do you agree that Big Data will result in rise of Predictive Enterprise Solutions? Please do share your thoughts.

Big Data: Will Open Source Software Challenge BI & Analytics Software Vendors

Predictive Analytics has been billed as the next big thing for almost fifteen years, but hasn’t gained mass acceptance so far the way ERP and CRM solutions have. One of the main reason for this is the high upfront investment required in Software, Hardware and Talent for implementing a Predictive Analytics solution.

As a result, only a handful of very large enterprises such as mega banks or top telecom companies have made the required investments and have benefited from power of Predictive Modeling and advanced Statistical techniques that are in existence for well over five decades. Most of the other companies have not been able to levarage power of business analytics as they cannot afford investing in specialized harware, database and BI/Analytics software applications being marketed by enterprise software vendors such as SAS and Teradata.

Well, this is about to change – thanks to technologies such as Apache Hadoop (which supports Big Data distributed applications under a free license), HBase (an open source, non-relational/distributed database) and the freely available R programming language (which is part of the GNU project). Using R, HBase and Hadoop, it is possible to build cost-effective and scalable Big Data Analytics solutions that match or even exceed the functionality offered by costly proprietary solutions from leading BI/Analytics software vendors at a fraction of the cost. And since R programming language is a freely available Open Source Software, users can leverage work done by others for specific analytics functionality and don’t have to re-invent wheel by rewriting the code. This reduces cost of developing analytics solution significantly.

Established BI and Analytics software vendors have no option other than offering their solution under SaaS (Software as a Service) model so that it is cost effective for their customers to implement analytics solution without requiring large upfront investment. This is all the more important for Big Data as the field is evolving rapidly. And if any BI or Analytics software vendor fails to adapt to this changing technological environment, they risk losing their market share.

 

Big Data Analytics a Key Enabler for Social CRM – Airlines Case Study

Big Data Analytics is a hot topic of discussion these days. But many feel that it is more of a “hype” and less of substance. In my opinion, Big Data Analytics is the real deal and if used correctly, can deliver great business results at a fraction of cost compared to other alternatives.

Here’s a simple yet very effective example of using analytics for understanding consumer attitudes towards airlines in real time. In this study, Jeffrey Breen has used the R programming language to analyze consumer sentiments about major U.S. airlines expressed on Twitter. For more on the methodology and results, see the following embedded Slideshare presentation (if you cannot see the embedded file, click here to view it on Slideshare website).

As you can see on slide no. 27, the twitter sentiment scores obtained  for many of the airlines are “comparable” to results of The American Customer Satisfaction Index (ACSI). What is important to note here is that by analyzing few tweets using freely available R programming language (which is part of the GNU project), it is possible to achieve results similar to that of an elaborate and expensive market research study such as ACSI at a fraction of a cost, that too in real time. Isn’t that a game changer?

Now imagine using Big Data Analytics to analyze Social Media data – not only text data but also pictures, video, GPS , Social Graph/Social Media linkages and using that information for engaging customers on Social Media channels. That in my opinion is the ‘holy grail’ (if I may use that term) of Social CRM.

In one of my earlier post, I have defined Social CRM as the business strategy of engaging customers through Social Media with goal of building trust and brand loyalty. Big data analytics is a key enabler for engaging customers on social media channels for building trust and loyalty (Social CRM). What do you think?

Why Big Data Analytics is The Next Big Opportunity for Outsourcing Companies

Big Data Analytics is making big headlines these days. Just check out a few from recent past:

So what is Big Data and why it is in the news so much these days?

According to Philip Russom, Director of TDWI Research, Big Data has three defining attributes – three Vs as he calls them. They are Data Volume, Data Variety and Data Velocity and together they constitute a comprehensive definition of Big Data. So Big Data is not just about Data Volume, but also the variety of data (mostly unstructured) and the velocity with which the data is generated and need to be analyzed. (for more, check out following posts by Philip Russom and the TDWI Best Practices Report titled Big Data Analytics):

Given three Vs of Big Data, ‘traditional’ data storage, retrieval and analytics methodologies are no longer going to work. Cloud Computing is going to play a key role when it comes to Big Data Management and Analytics. And here in lies the opportunity for Outsourcing companies.

Traditionally, data collected by organizations is ‘safely’ stored in massive relational database accessible to only few within the organization and requires elaborate infrastructure both in terms of hardware and software for storage, retrieval and reporting/analytics. In such an environment, it is not possible to easily outsource Data Analytics function/processes alone given the heavy investments made in terms of hardware and software.

Because of the business requirement of analyzing vast amount of ever changing structured and unstructured Big Data almost instantaneously, companies will be hard pressed to do this on their own. But given the fact that Big Data stored in cloud can be accessed from anywhere the internet is available and can be analysed almost instantaneously by third party service providers,  outsourcing companies can offer to their clients value added services in the area of Big Data analytics without heavy investments on the part of clients in specialized hardware and software as was the case with ‘traditional’ data analytics. This will bring down significantly costs (especially fixed costs) associated with building and maintaining analytics infrastructure and solution center.

Just to give an example in the area of Social CRM, Social Media has empowered customers like never before as they can discuss about brands/products on Social Media channels. The best any marketer can do is to Listen, Learn and Engage customers. Given the three Vs of Big ‘Social’ data and the fact that most of the user generated content resides in the cloud, outsourcing companies can offer cost effective analytics solution to their clients to enable them effectively engage their customers/prospects in real time.

What do you think? Do you agree that Big Data Analytics is The Next Big Opportunity for Outsourcing Companies? Look forward to hearing your thoughts and comments:

  • About Dr. Harish Kotadia


    That's me with photo gear,  taking snaps of Texas wild flowers. #texas

  • Dr. Harish Kotadia

  • Dr. Harish Kotadia is an industry recognized thought leader on Big Data and Analytics with more than fifteen years' experience as a hands-on Big Data, Analytics and BI Program/Project Manager implementing Enterprise Solutions for Fortune 500 clients in the US.

    He also has five years' work experience as a Research Executive in Marketing Research and Consulting industry working for leading MR organizations such as Gallup.

    Dr. Harish Kotadia's educational qualification includes Ph.D. in Marketing Management. Subject of his doctoral thesis was Customer Satisfaction and it involved building a statistical model for predicting satisfaction of clients with services of their ad agency.

    His educational qualification also includes M.B.A. and B.B.A. with specialization in Marketing Management and Diploma in Computer Applications.

    Dr. Harish Kotadia currently works as Principal Data Scientist and Client Partner, Big Data and Analytics at a Global Consulting Company. Views and opinion expressed in this blog are his own.



  • Subscribe to this blog via RSS or Email


     Subscribe in a reader

    Enter your email address:

    Delivered by FeedBurner

  • Search this blog:




  • Tag Cloud

  • Calendar of Blog Posts:

  • August 2015
    S M T W T F S
    « Mar    
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    3031  


  • © 2014 Harish Kotadia. All Rights Reserved.
  • Harish Kotadia's Flickr Photos


    By Erik Rasmussen