New Research by the BCG Henderson Institute Examines Impact of GenAI on Consultants’ Performance of Data Science Tasks Beyond Their Existing Capabilities
Generative AI (GenAI) is creating a new type of knowledge worker who can code faster and summarize documents instantly. But can the tool also enable people to meet the shifting demands of their roles? A new scientific experiment conducted by the BCG Henderson Institute in collaboration with BCG X and Emma Wiles from Boston University, as described in GenAI Doesn’t Just Increase Productivity. It Expands Capabilities, examines what happens when, instead of using GenAI to improve performance within their current skillset, people use the technology to complete tasks beyond their existing capabilities.
In the experiment, 480 BCG consultants completed two of three short tasks that mimic the daily activities of a data scientist: writing Python code to merge and clean datasets, building a predictive model, and validating ChatGPT-generated statistical analyses. These tasks were designed to present a significant challenge for any consultant and could not be fully automated by the GenAI tool (Enterprise ChatGPT with GPT-4 and its Advanced Data Analysis Feature). To help evaluate participants’ performance, their results were compared with those of 44 BCG data scientists who worked without the assistance of GenAI.
“Our findings suggest that GenAI-augmented workers can adeptly handle new tasks beyond their existing skills in fields that are in the tool’s capabilities,” said Dan Sack, a BCG managing director and partner, and coauthor of the study. “Executives need to be ready for this future, redefining expertise and identifying the skills to grow and retain their talent for the long term.”
Catch more HRTech Insights: HRTech Interview with Sam Naficy, Chairman and CEO at Prodoscore
Instant Aptitude Expansion for New Tasks
When using GenAI, the consultants in the study were able to instantly expand their aptitude for new tasks. Even when they had no experience in coding or statistics, consultants with access to GenAI were able to write code, appropriately apply machine learning models, and correct erroneous statistical processes. The biggest skill expansion was observed in coding, where participants were tested on their ability to write code in Python, a common programming language by data scientists. Participants who used GenAI achieved an average score equivalent to 86% of the benchmark set by data scientists, a 49 percentage point improvement over participants not using GenAI. The GenAI-augmented group also finished the task roughly 10% faster than the data scientists.
GenAI as a Powerful Brainstorming Partner
For the predictive analytics task, the experiment participants encountered a significant challenge: neither they nor the GenAI tool had a high level of proficiency in this area. Predictive analytics was the task which the GenAI-augmented consultant was least likely to perform on par with a data scientist, regardless of previous experience in coding or statistics, as the GenAI tool is likely to misunderstand the reliability prompt without trial and error or rephrasing of the question. As a result, participants with access to GenAI were more likely to be led astray than their nonaugmented counterparts.
With the support of GenAI, participants were able to brainstorm with the tool, combining their knowledge with GenAI’s knowledge to discover new modelling techniques and identify the correct steps to solve the problem successfully. The GenAI-augmented participants were 15 percentage points more likely to select and appropriately apply machine-learning methods than their counterparts who did not have access to GenAI.
“Doing GenAI” Doesn’t Mean Learning to Do
Reskilling is defined as individuals gaining new capabilities or knowledge that enables them to move into a new job or industry. The study revealed that GenAI-augmented workers were in a sense “reskilled” in that they gained new capabilities that were beyond what either the human or GenAI could do on their own. But GenAI was only an exoskeleton; the humans alone were not intrinsically reskilled, because “doing” with GenAI does not immediately nor inherently mean “learning to do.”
While each participant was assigned just two of the three tasks in the experiment, everyone was given a final assessment with questions related to all three tasks to test how much they learned. Everyone was asked a coding syntax question, even though not all participants had done a coding task. Surprisingly, those who did this task scored the same as those who didn’t, indicating that performing the data science tasks did not increase their knowledge. It should be noted that the participants were not informed that they would be tested, and it is likely that with repetition and intention, learning would happen.
In addition, GenAI-augmented participants with moderate coding experience performed 10 to 20 percentage points better on all three tasks than their peers who self-identified as novices, even when coding was not involved. In fact, those with moderate coding experience were fully on par with data scientists for two of the three tasks—one of which had zero coding involved.
“As a parent, I often get asked what kids should study, and it’s something I think about when it comes to my own children,” said Sack. “This research has reinforced my belief that learning to code holds significant value, even if the prevailing opinion suggests that coding might be a thing of the past. It’s the engineering mindset that coding helps develop—like the ability to break down complex problems into manageable parts that can be efficiently tested and refined—that truly matters.”
Read More on Hrtech : HRTech Interview with Louise Willoughby, Chief People Officer at Beekeeper
[To share your insights with us, please write to psen@itechseries.com ]