The concept of big data did not emerge overnight; it synthesizes advancements from various technological domains such as data computing and data storage. The application of big data in various industries is also profoundly changing their development status. Such applications are commonplace in both industry and academia, with numerous “fancy words” often employed. In this blog, I try to address three key questions regarding the application of data science in procurement from a technical perspective, while avoiding the excessive use of “fancy words”. The three questions are: Where are we now? How will data science impact the field? And where are we going in the future? While it may be impossible to provide perfect answers to these questions in a short essay within a short time, I believe that the question itself, and the insights or possible discussion this essay may bring, could be more important than some of the specific answers I noted today.
1.Where we stand now?
It is almost universally agreed that we are living in an era of information explosion: Companies are nearly overwhelmed with massive amounts of data concerning customers, suppliers, and potential markets due to new digital technologies. This information is a source of power that remains useless if it is not properly exploited. On the other hand, some surveys on senior executives also found that, while almost 100 percent of the sample understood how big data could benefit their supply chain management, only few of them implemented it in practice. Most data and analytics projects in procurement today are small data projects which are either descriptive or diagnostic analytics. The level of digitization in this industry is lagging behind comparing to some of the highly digitalized industry such as banking. We can describe this phenomenon as a “chimney-style” system, where
different business areas and modules are independent systems without integration. Here we list some of the small independent data projects, including increasing accuracy in the demand forecasting (forecasting), fraud detection during bidding process (classification), data visibility during the procurement process and so on. These projects all provide clues to answer the second question:
2. How will big data affect each process of procurement?
When you view big data as a toolbox, for well-defined problems such as analyzing and predicting price changes of certain raw materials, forecasting the demand for specific materials by specific companies, and expenditure analysis and management, you can select appropriate tools from the toolbox to solve these problems. This represents the concrete realization of data-driven decision-making in various aspects of this field and constitutes the low hanging fruits in the digitalization process of the procurement industry. But even these lowing hand fruits can have a fundamental impact on procurement organization. And based on these, there are still many unexploited opportunities out there.
3. Where are we going?
After going through the low-hanging fruits, we might be more interested in knowing where the higher fruits are, and what the challenges are there in this journey. To answer this question, we try to firstly redefine ‘big data’ – from a more comprehensive perspective. In fact, how to properly define ‘big data’ in business management is an important research direction in the intersection of data science and business management. This also implies the future digital development direction of these fields. Based on the classic Laney’s (2001) 3Vs model: Volume, Velocity, Variety framework. Showing below is one of my favourite versions: Krishnan (2013).
“Big Data analytics can be defined as the combination of traditional analytics and data mining techniques along with large volumes of data to create a foundational platform to analyze, model, and predict the behavior of customers, markets, products, services, and the competition, thereby enabling an outcomes-based strategy precisely tailored to meet the needs of the enterprise for that market and customer segment.”
This is the most comprehensive version of the integration of data science and application that I have seen so far. In this definition, we can also see the future development directions and technical challenges:
3.1 From Combination to Integration:
The integration of data based on big data models should not simply be the stacking of data and models. Compared to the previous ” chimney-style” system, where each department operated independently, data integration across various business processes allows for the extraction of more information from higher-dimensional data. Currently the banking industry has good applications in this area. For example, by combining financial transaction data and customer information to analyze customer credit and using machine learning algorithms to analyze customer transactions, banks can ultimately form a comprehensive customer credit scoring system.
Effectively integrating data resources across systems to achieve system convergence and development will be a new challenge for many technology-driven insurance companies. Fortunately, we already have a tool in pocket to have us to get this done. The tool is called Blockchain- it has had a breakthrough impact on project management, cross-platform integration of resources, and ensuring the security of information flow. For example, Guardtime, a blockchain technology company, collaborated with Sinolink Worldwide Holdings Ltd. (0732.HK) on a blockchain-based marine insurance platform project, integrating different data and processes to reduce data inconsistency issues and decrease error rates.
3.2 Causality vs. Correlation:
In the era of data explosion, one trap that data science often touches upon in applied fields is the so-called “overfitting.” Data mining aims to discover and utilize information beyond traditional human experience, rather than just training models on a high-dimensional, highly integrated, ex post dataset and playing an overfitting game by using the ex post data. Perhaps you have heard many techniques to avoid overfitting’ during the model training process, but it seems that few truly explore how to avoid overfitting from a fundamental perspective. In a mature data-driven decision-making industry, the most typical example being quantitative finance, we might mitigate ‘overfitting’ during the model training process through engineering tricks, but we should realize a basic fact: In these mature fields, both the depth and breadth of the qualified data may far exceed those of emerging data science application industries, such as procurement and insurance industry.
Although this perspective is rarely discussed seriously, I think it is crucial. Current machine learning techniques largely rely on mining data correlations, which may lead to a series of deficiencies in model stability, interpretability, and fairness. These deficiencies will be further amplified in emerging applications such as the procurement industry due to the limitations in data depth and breadth. This makes integrating causality into the machine learning framework particularly important. Causality represents a higher level of knowledge discovery compared to correlation, and the process of uncovering causality is more complex than that of finding correlations. Therefore, this constitutes another challenge for the application of data science in emerging industries.
