Exponential Data Growth
More Data, Less Gains
Data is growing at an exponential rate, but computer performance and productivity gains have plateaued.
In 1965 Gordon Moore, the co-founder of Intel, wrote “The complexity for minimum component costs has increased at a rate of roughly a factor of two per year. Certainly over the short term this rate can be expected to continue, if not to increase.” This statement was simplified by others and eventually coined as Moore’s Law, meaning that the number of transistors on a microchip doubles approximately every two years. This law has mostly held true for over half a century. More impressive is that the economic forces driving this growth have continued. With this change in technology, we are experiencing what seems to be exponential growth in business communication and the data managed. Below are some charts that are a testament to the exponential increase in computation and data.
Ray Kurzweil has written extensively about Moore’s Law and how this technological growth will continue to affect society. Kurzweil has coined his own term for this exponential growth of the different types of technology we are seeing as “The Law of Accelerating Returns.” This idea has been associated with a lot of IT hype. Believing too strongly in tech hype has created at least a few market bubbles… However, Kurzweil’s predictions have also held up quite well over the decades. A key aspect to why many of these exponential growth rates have continued rather than following an S-curve or logistic curve trajectory is because these IT evolutions are fueled by just information and time. Digital information takes up very few resources. The abundance of data storage space has allowed the exponential trend to continue so far without hitting a wall of resource limitations.
IT has enabled not only the formalization and tracking of operations, but more recently the extreme automation of evermore complex processes. As IT systems and the amount of cheap storage have advanced, the growth of digital data has followed. This lockstep improvement of computation and data storage requires businesses to keep up with the competition that is using more efficient and creative processes. However, the bottleneck that is continuing to grow in the industry is the communication gap between different data sources and organizations. If businesses don't simplify their data management now, it's only going to get more difficult to connect different systems and automate processes.
Hoping that artificial general intelligence (AGI) will arrive soon and solve our problems is unlikely. Since around 2005 multiple measures of computational performance have plateaued (see chart below). The maximum performance for a single CPU has not increased much since then. Actual improvement is being created by providing additional core units of CPU parts in a reassembled configuration to process more applications at the same time. This architectural improvement comes with the additional transistor count (e.g. cores and adaptive circuits) without exceeding physical limits of thermal power at optimum frequency levels. Facing this power constraint, the number of cores that provide additional performance will likely be limited by the end of 2030. This power limitation will result in a requirement for even more distributed computing.
However, new paradigms in computational technology are continuing the trend of exponential growth in some areas. 3D integrated circuits are the most obvious recent advance that allows for a continuation of Moore’s Law. Businesses opting to design specialized chips and continuously innovate manufacturing processes are speeding up this feedback loop. Furthermore, research on quantum computation is opening possibilities for materials science that will allow for even smaller technology. The number of qubits of various types in individual quantum computers is growing at an exponential rate and following a trend that some are calling Rose’s or Neven’s Law, displayed below. Eventually, the number of qubits along with a low enough error rate will allow quantum computing to be useful and sustainable. The increase in the number of the endpoints on the cloud systems with these newer and more powerful computers will continue to follow Kurzweil’s Law of Accelerating Returns far into the future. Therefore, data storage and processing will continue to increase exponentially.
These exponential trends have resulted in plateauing costs for computation and storage (see charts below). At the same time, productivity growth (measured through GDP) has shown a downward trend. This stabilization of IT and automation benefits is likely due to the lag in modernizing business automation and ways of working. The lack of sufficient data management and change management seems to be the largest barrier that companies are facing. Even if the positive effects from IT improvements are making less of an impact on a regular basis, the reduction in computational costs will continue to help businesses in the near term. Leveraging existing technologies and shortening software production cycles will continue to yield significant results.
The path to AGI will be a long one. While most of us continue to operate within the resource constraint of electrical power from a typical wall outlet, we will see even more migration to intensive cloud computing for our applications. In previous decades the cost of computer circuits was mostly determined by R&D investment and manufacturing technology implementation. Now we have reached a stage where semiconductor circuits are on average costing $500 at initial production, regardless of function or manufacturing platform. Commoditization of a larger percentage of the technology industry is pushing prices even closer to just the cost of the raw materials. So for most new product offerings, the difference will be made through better programming and more distributed computing.
The low hanging fruit for most businesses is implementing better data management processes. Sure, available software and hardware will continue to improve and help us. Yet, what will matter is how we leverage this technology with better communication and systems. More emphasis on business process design and implementation is necessary to realize improved performance. These changes will continue to cause significant disruptions in industry. Banking on quantum computing or AGI to solve problems or do our jobs for us is not realistic. In the face of exponential data growth, it's up to today's business leaders to leverage that data to innovate and create even more value.