The demands of the rapidly changing business environment and the advancement of the Al age is already beginning to bring in significant disruption to the data talent landscape as well. And therefore, I expect a significant overhaul where a few roles become more prominent, and many undergo significant change and become more specialized or narrower.
As we discussed, value is generated at each stage of the data management value chain. But in the Al age, the two ends of the value chain, the ‘data’ and customer/business ‘impact’, play a critical role in shaping the talent landscape. Data, which is growing in complexity and scale and fuels the massive AI models, holds utmost importance in terms of quality, relevance and proper utilization. So, the role of data professionals in handling such complex data landscapes is naturally expanding. On the other hand, as the importance of leveraging data effectively in driving business outcomes has increased tremendously, it has become increasingly important for data professionals to expand their capabilities beyond core data skills to become more business outcome focused.
I foresee three key trends that will shape the data talent landscape:
- Traditional data science roles are shrinking, becoming more specialized Once dubbed the ‘sexiest job of the twenty-first century’, data science has now become an integral part of data-driven organizations, and will continue to be in high demand especially with the rising importance of Gen AI. Data scientist roles are expected to increase by 35 per cent from 2022-23, making it one of the fastest-growing professions in the United States. While I acknowledge that this trend is likely to continue, the conventional skill set associated with the data scientist role may become less relevant, paving the way for a fresh definition of what it means to be a data scientist. Here is how things are rapidly evolving:
- Automation and Al advancements: The progress of Al and automation is causing many of data scientist’s traditional data preparation tasks to shrink. Tasks like data cleaning, data processing and data preparation and feature creation can now be automated through tools like AutoML (automated machine learning). Approximately 80 per cent or more of a data scientist’s job involves preparing data for analysis and this can be significantly automated now. The emergence of low-code and no-code platforms, which are easy to use, has simplified the data preparation process, making it more accessible and widely adopted, freeing up a data scientist’s time to focus more on generating relevant insights.
- Pre-built models on cloud: Hyperscalers like AWS, Azure and GCP boast rich libraries of pre-built analytics models that are easily deployable with minimal customization and which streamline the process of generating insights. This significantly cuts down on the time-consuming model development process, which has traditionally been the ‘bread and butter’ of data scientists. As a result, these traditionally specialized tasks can now be done with minimal expertise.
The above trends are leading to the following changes in the role of data scientists:
- The rise of ‘citizen data scientists’: Traditionally, data scientists were primarily responsible for statistical modelling and building various data models. The democratization of data and AL/ML automation tools has led to the rise of ‘citizen data scientists’—individuals with limited formal training in advanced analytics, statistics or related disciplines who perform data-related tasks. The availability of various prepackaged and user-friendly data analysis and visualization tools has made it easier for non-technical users to work with data. As a result, this role is now often played by business analysts but also many other profiles. They are expected to be more data competent and to handle tasks ranging from defining data requirements to building simple data models, which were previously the domain of data scientists.
- High focus on specialized roles: The need for a generalist data scientist is shrinking and their role is becoming more nuanced or specialized. In the Al age and Gen AI in particular, while consumption of models has become easier, the creation of these models is way more complex and requires sophisticated skills. These models require, dealing with advanced algorithms like LLM, artificial neural networks (ANN) and NLP, among others.
- Business-related aspects are gaining prominence
I believe that with the growing involvement of data professionals in driving business outcomes, they require capabilities beyond their core data skills. First, they must possess deeper domain knowledge related to the industry or functional domain they are aligned to, which would enable them to effectively connect data insights to real-world business problems and opportunities.
Without a solid understanding of the specific business domain, data professionals may struggle to connect with the business problems and comprehend their challenges effectively. It would impact their ability to ask the right questions. Second, they must possess strong problem-solving skills, essential to identify and address complex data-related challenges. This is because in addition to analysing data, they are also expected to propose solutions to business problems, optimizing processes and uncovering opportunities for growth.
Another critical skill that has become paramount for a data professional is the art of storytelling. Data professionals are often tasked with presenting their findings to nontechnical stakeholders; this has given rise to the importance of data storytellers or data translators. Strong storytelling and communication skills allow data professionals to convey complex data insights in a compelling and understandable manner. Organizations like McKinsey recognized the importance of data storytellers or data translators early on and took proactive steps to address the need. McKinsey established an academy internally in 2017 to train 1000 individuals specifically for these roles.
- Rising importance of data engineers and data architects
I also believe that the role of data engineers will expand substantially, primarily driven by the exponential growth of data and the engineering complexity associated with it. Add to that, the rapid advancement of AI technologies. This demand for data engineers is poised to cut across industries as organizations increasingly recognize the importance of leveraging data for decision-making. Likewise, the role of data architects will gain even greater significance in the Al age. They will be instrumental in designing and implementing the complex and scalable data infrastructure required to support AI, and now Gen AI as well. The expertise of data architects, in areas such as data modelling, integration, governance and system design becomes vital for managing the complexities inherent in the ever-expanding data ecosystem.
This excerpt from Mastering The Data Paradox: The Key To Winning In The AI Age by Nitin Seth has been published with permission from Penguin Random House India.