About 90% of the data available in a company is not used. In many cases, the existence of this data is not even known. In doing so, it can be of central importance for a company to take care of this “dark data”: important information might be hidden that can increase competitiveness, generate sales potential and reduce risks.

Causes

There are various reasons for the emergence of dark data: In the course of digitization, the number of data sources and the arising amount of data have increased significantly. At the same time, the cost of data storage has fallen sharply. For safety’s sake, many companies store all sorts of data for later (but never performed) analyzes. Data and documents are often copied, modified, stored in different silos and versions and are only available to individual employees or departments. Some data is incorrect, others become obsolete and are never deleted. In the worst case, however, data that can bring real added value is simply ignored – whether because it’s not known, cannot be retrieved, has no access, or lacks the resources and capabilities to analyze it.

Discover, classify, use and manage dark data

In order to be able to lift the hidden data treasures, one must first of all find them – either by making a comprehensive inventory of all existing data in the company (= data assessment) or by searching for specific information with appropriate tools and methods of data retrieval and information retrieval (only possible if the necessary access is provided).

Next important step is to separate the wheat from the chaff, or more precisely: to distinguish ROT data from business-relevant data. “ROT” is the abbreviation for “redundant, obsolete, trivial” and denotes, for example, outdated, damaged or incorrect data without added value. These are primarily an expense factor and should be deleted. In addition, appropriate data management should prevent the future accumulation of ROT data.

Dark data is business-relevant when it comes to risks or opportunities. Risks arise, for example, when “forgotten” data is subject to lower security standards and becomes the target of hacker attacks. If the hacked data is personal information, it can quickly become very expensive for a business. In the light of GDPR personal data should be easy to find, modify, export and delete (right of access, correction, transfer and deletion).

In turn, data can offer numerous opportunities: recordings of customer calls or e-mail complaints can provide important insights into pain points, customer sentiment or optimization potential of products and services. Log files provide insights to the behavior of website visitors and ways to improve performance or conversion. Geospatial data can be used to reconstruct the customer’s movement behavior and make it usable for further business planning (geo-tagging).

For the evaluation of the varied, structured and unstructured data, for example, data mining and text mining methods can be used in addition to classical statistical methods. In this way, patterns (classifications, segmentation, forecasts, dependency analyzes, variance analyzes) and important topics, moods and trends can be identified and subsequently used for marketing, sales and service.

Of course, a company can approach its dark data within a project first. In the long term, however, the use of (dark) data should be strategically planned and established firmly. This requires a comprehensive data strategy tailored to the respective business strategy, which sets the guidelines for data management, data quality assurance, data provision and data usage and ensures that IT architecture, business organization and data value creation work together seamlessly.

If you have any questions or need help raising your hidden data resources, please do not hesitate to contact us via our contact form or write an e-mail to info@thaltegos.de.