Big data challenges impacting data-driven business goals

IT leaders need to understand the opportunities big data presents so they can overcome the various challenges and remain competitive in the growing data-centric economy.

big data abstract visualization

The exponential explosion of digital data has forced researchers to find new ways of seeing and analyzing the world. It's about discovering new orders of magnitude for capturing, searching, sharing, storing, analyzing and presenting data. That’s how "big data" was born. Big data is a concept for storing a huge amount of information on a digital basis.

Big data refers to a very large set of data that no conventional database management or information management tool can really work. In fact, we produce about 2.5 trillion bytes of data every day. This amount of data come from different platforms: messages we send, videos we publish, weather information, GPS signals, transactional online shopping records and more. These data are called big data or massive volumes of data. The Web giants, first and foremost Yahoo (also Facebook and Google), were the first to deploy this type of technology.

Though there is no specific or universal definition of big data. Being a complex term, its definition varies according to the communities that are interested, user, or service provider. A transdisciplinary approach allows understanding the behavior of the different players: the designers and suppliers of tools (the computer scientists), the categories of users (managers, business owners, political decision-makers, and researchers), as well as professionals.

Big data is a dual technical system. Indeed, it has its benefits and challenges. The arrival of big data is now presented by many articles as a new industrial revolution similar to the discovery of steam (early 19th century), electricity (late 19th century) and computer science (late 20th century). Others describe this phenomenon as the last stage of the third industrial revolution, which is, in fact, the “information age”. In any case, big data is considered a source of profound disruption in society.

Big data is becoming more popular among businesses in all industries and undertaking a big data project is not easy. According to a study conducted by NewVantage Partners, 95% of the Fortune 1000 entrepreneurs surveyed have undertaken a big data project in the past five years, but only 48.4% have managed to benefit from these projects.

Below are some of the big data challenges businesses encounter:

Managing data growth

Clearly, one of the biggest big data challenges to overcome is to store and analyze all the information. According to the "Digital Universe" report, IDC estimates that the amount of information stored in computer systems around the world doubles every two years. Most of this data is unstructured, which means that it is not stored in a database. Photos, documents, videos, and audio files are difficult to analyze.

To overcome this challenge, companies can use different technologies to manage the constant increase in data. In terms of storage, converged and hyper-converged infrastructures, as well as software-defined storage, are proving to make things easy to scale hardware. Technologies such as compression, deduplication, and tiering also reduce the space required and the costs for storing big data. With regards to management and analysis, companies can use tools such as NoSQL, Hadoop, Spark and other big data analytical software, as well as business intelligence software, AI and machine learning to get the insights they need.

Generate insights quickly

Businesses do not just want to store the big data they generate. They are more interested in using big data to achieve their goals. According to the study conducted by NewVantage Partners, the main objectives associated with big data projects are the reduction of expenses, the implementation of a data-driven culture, innovation, the acceleration of the deployment of new capabilities and services and the launch of new products and services. These different goals can make companies more competitive, but they need to get insights and exploit them quickly.

To help them get to this speed, companies can use a new generation of analytical tools that significantly reduce the time needed to generate reports. They can invest heavily in analytical tools that will help them get results in real time. By this, they can respond to developments in the marketplace as fast as possible.

Recruiting big data talent

To develop and manage applications that generate insights, companies need professionals with big data skills. In fact, the demand for big data experts has risen dramatically, along with the salaries offered by companies.

To deal with the lack of big data talent available, companies can use one of several options available. They can increase their budgets and efforts in recruitment and retention. Other options to consider are training their current employees to learn and master big data – developing big data talent from within. Finally, many firms are turning to technology. They purchase self-service analytical solutions or Machine Learning software designed for use by professionals without a degree in data science. These tools can help companies overcome their big data challenges and reach their goals without even hiring qualified experts.

Integrating diversified big data sources

The wide variety of data makes integration one of the biggest big data challenges. Indeed, the data come from different sources: business applications, social networks, emails, employee documents...combining all these data harmoniously and using them to create reports, and for those advanced users, data-driven insights and business decision support solutions, can be very difficult. To address this problem, various vendors offer integration tools designed to facilitate the process. However, many companies admit that they are yet to overcome this challenge.

Data validation

Data validation is also one of the major challenges of big data. Many companies receive similar data from different systems, and this data is sometimes contradictory. For example, an e-commerce system may have a certain level of daily sales, while an Enterprise Resource Planning (ERP) system may have a slightly different level.

In order to harmonize this data, companies must use data governance (you can do internal linking here for SEO). Data governance also presents various challenges, and it is in fact, the fastest-growing area of concern cited by respondents according to AtScale 2016 "Big Data Maturity Survey."

Solving data governance challenges is usually not easy. It requires a combination of technology and policy change. However, some attempting includes allocating a group of people to monitor data and defining rules and procedures. Another option is to invest in data management solutions designed to simplify data governance geared towards big data accuracy and storage.

Securing big data

Security is also an important concern in the field of big data. Business data can be attractive to hackers. However, according to a study by IDG, only 39% of companies use additional security measures for their data repositories. Some of the most popular additional measures include access and identity control, data encryption, and data segregation.

Organizational resistance

In addition to the technology aspects of big data challenges, employees can also represent a big data challenge. Among the main challenges encounter by companies trying to launch a big data project, the three main problems are insufficient organizational alignment, lack of understanding on the part of managers, lack of understanding or business resistance.

To solve this challenge, it is, therefore, necessary to convince business leaders of the usefulness of big data and to appoint a Chief Data Officer.

It is also important that executives, directors and managers understand the opportunities big data presents so they can overcome the various challenges and remain competitive in the growing data-centric economy.

This story, "Big data challenges impacting data-driven business goals" was originally published by CIO.