The Promise and Progress of Generative AI

[this post originally appeared on Gradient Flow.]

One year after ChatGPT’s launch, generative AI is gaining mainstream traction. Large language models (LLMs) like GPT-4 are being adopted across diverse real-world applications, from chatbots to computer programming to medicine and law. The speed of adoption has been remarkable – shortly after the release of GPT-4, chatbots leveraging its capabilities were publicly available. Thanks to the generalizability of LLMs, use of generative AI has been extremely rapid. The future promises even more powerful frontier models. But even now, evidence shows these technologies are already having a transformative impact.

LLMs are seeing significant adoption across sectors, thanks to their few-shot and zero-shot learning abilities. Applications span chatbots, computer programming, medicine, law, robotics, and more, with over 100 papers in 2022 detailing novel uses. In certain specialized tasks, LLMs can match or exceed human-level performance, like in medicine and law.

Recent months have seen extremely rapid diffusion of LLMs into applications, impacting industries through new use cases. Content creation has exploded, with tools like Anthropic’s Claude and OpenAI’s DALL-E generating marketing and advertising materials. AI chatbots and custom LLMs can now enable human-like conversations, opening up new possibilities for customer service and information search. Process automation using AI is eliminating repetitive workflows in areas like contract analysis (LawGeex), radiology reporting (Nuance), and customer support interactions (Infinity). In drug discovery, companies like BenevolentAI and Exscientia are applying generative AI to accelerate development of new medicines, with the potential to greatly shorten R&D timelines. Concerns persist around content authenticity, chatbot consistency, over-automation of jobs, and the unproven track record and robustness of Generative AI. Nonetheless, real-world AI adoption continues rapidly.

The View of Executives and Managers

A recent Fortune/Deloitte CEO Survey found widespread CEO interest in generative AI’s potential. 79% of CEOs surveyed believe the technology will increase efficiency, and over half think it will provide new growth opportunities. More than half reported they are actively evaluating and experimenting with generative AI. Perhaps most notably, over a third indicated they have already started implementation to some degree, demonstrating that for many companies it is rapidly moving beyond experimentation into active use.

Other surveys confirm substantial enthusiasm and growing investment in generative AI among enterprise leaders. Nearly half of executives in a PwC survey plan to invest in the technology, with over two-thirds believing it will significantly impact their business model, revenue and growth. Meanwhile, a survey by the AI Infrastructure Alliance found over half of companies expect double-digit percentage revenue increases in the coming year from their AI investments, pointing to bullish forecasts around technologies like generative AI. These results make clear that business leaders see major potential in generative AI to transform operations and boost bottom lines.

While people generally are confident that generative AI will affect their businesses, how it does so is less clear. A recent MIT survey shows business leaders agree generative AI will impact their companies. However, views differ on how. The largest share (24%) foresees workforce reductions. Others see threats to offerings, potential for new sectors, or needs to recalibrate staffing. Despite uncertainties, most avoid doom-laden views, instead taking a thoughtful approach.

Measuring the Impact of Generative AI 

Advances in generative AI are positively impacting companies well-positioned to adopt the technologies, according to a UCLA and USC study. The study found that after ChatGPT’s release, stocks of firms with high workforce exposure to generative AI outperformed less exposed firms, with over 0.4% higher daily returns. This effect was seen both across and within industries. As AI becomes an increasingly frequent earnings call topic, the study validates a new measure of workforce exposure to generative AI. The arrival of LLMs represents a major technology shock affecting firm values and creating winners and losers. This helps explain why executives are excited about generative AI’s transformative potenti

l.

Generative AI is rapidly gaining traction among executives and managers due to early studies demonstrating its benefits. One such study from M.I.T. examined the productivity effects of ChatGPT for 444 college-educated professionals performing mid-level writing tasks in various occupations, including marketing, consulting, and human resources. The study found that access to ChatGPT led to substantial productivity gains, with output quality increasing by 0.4 standard deviations and time taken decreasing by 0.8 standard deviations. Notably, these gains benefited lower performers more. ChatGPT increased the output quality of low-ability workers while reducing their time spent on tasks, and allowed high-ability workers to maintain quality while working faster. Overall, ChatGPT compressed the productivity distribution, reducing inequality between workers.

Another real-world study done by Stanford and M.I.T. researchers looked at the impact of a generative AI conversational assistant for over 5,000 customer service agents. This field study found that access to the AI tool, which combines GPT with additional models fine-tuned on past conversations to provide real-time suggestions and documentation links, increased agent productivity by 14% on average, measured by resolved issues per hour. The largest gains went to novice and low-skilled workers, whose productivity increased by 35%, while highly skilled workers saw minimal impact. Agents who followed the AI’s recommendations more closely realized bigger productivity gains. The study also found the AI improved customer sentiment, increased employee retention, and may have led to skills development, as agents performed better even when the AI was unexpectedly disabl

d.

Together, these two studies demonstrate that generative AI can provide substantial productivity gains that tend to benefit lower performers more. The technologies appear to disseminate best practices and accelerate development for newer workers, while allowing more experienced employees to focus their time on higher-value tasks. While questions remain about the long-term impacts on wages and employment, these studies give managers reason to be excited about potential productivity and retention benefits from judicious incorporation of generative AI. More study is needed, but the early results are promising for workers as well as businesses.

Frontier Models for Programming

AI coding assistants are already seeing widespread adoption by developers across all experience levels, contrary to popular perception. A recent Stack Overflow survey of 90,000 developers found that AI tools are widely used to write code, even among the most experienced coders.

An impressive 84% of developers with less than 1 year of experience reported using AI coding assistants, and usage rates barely declined for more experienced developers. Even among those with 6-10 years of experience, 86% said they use AI writing tools, and 88% of developers with over 20 years of experience are leveraging AI to help write code today.

The surge of AI in software development has been staggering – with nearly 300,000 distinct AI-related projects created on Replit in 2022 alone, growth accelerated to 160,000 projects in Q2 2023, a 34x increase year-over-year. New developer tools like LangChain and open source models are fueling an explosion of innovative applications, from autonomous agents writing their own code to AI avatars, that highlight the transformative potential of this technology.

Industry leaders are taking notice of these trends. In his annual letter, Microsoft CEO Satya Nadella highlighted examples of how AI coding tools like GitHub Copilot are supercharging developers. OpenAI research director Andrej Karpathy reported that Copilot was writing 80% of his code at 80% accuracy. Mercado Libre found that Copilot reduced their developers’ coding time by over 50%. And Microsoft’s own support agents use Copilot to resolve cases faster.

While AI coding assistants are certainly beneficial for new developers, their capabilities are clearly resonating across all experience levels. The data shows that these technologies have become integral for even the most seasoned technical professionals today. With broad adoption by the most expert coders, AI is already fundamentally transforming software development.

The Perspective of Chief Data Officers

Given their oversight of data strategy and governance within organizations, Chief Data Officers (CDOs) are poised to play a pivotal role in the adoption of generative AI. As organizations pursue digital transformation, generative AI offers a unique opportunity to streamline processes, enhance decision-making, and create innovative products and services.

A recent AWS Survey of over 300 CDOs worldwide provides valuable insights into their perspectives and preparations for generative AI. While adoption is still in its early stages, enthusiasm among CDOs is high, with 93% agreeing that data strategy is crucial to extracting the technology’s value

However, CDOs also face significant obstacles to fully leveraging generative AI. One of the biggest challenges is data quality. Custom large language models and retrieval-augmented generation applications are notoriously data-hungry, requiring high-quality datasets highly relevant to the task at hand.  However, many organizations struggle with data silos and quality issues. Nearly half of CDOs cited data quality, along with data privacy and security as top challenges.

Another concern is responsible AI guardrails. Generative AI models can be susceptible to bias and misinformation, which can have serious consequences. CDOs recognize the need for governance, and just under half of those surveyed pointed to responsible AI guardrails and data privacy/security as concerns.

Despite the challenges, CDOs are optimistic about the potential of generative AI to transform their businesses. Key data initiatives underway include more data integration, surveying data utility, and curating documents or text for domain-specific applications. Overall, the survey findings highlight the critical role CDOs play in driving generative AI adoption. As organizations work to transition generative AI from experiments to production systems, CDOs will be responsible for tackling data challenges, developing data strategies, and implementing responsible AI practices.

Generative AI Mainstreaming Accelerates

The momentum behind generative AI is evident. Beyond the examples discussed, trends in startup funding, job postings, patents, and more all point to rising mainstream interest. Generative AI startups attracted close to 5 times more investment in the first 9 months of 2023 compared to all of 2022, with $21.4 billion invested through September 30 versus $4.35 billion last year, PitchBook data shows. Job postings for roles like “AI specialist” grew over 50% year-over-year in 2022. And patents related to generative AI quadrupled between 2022 and 2023.

Across metrics, it’s clear generative AI represents a rapidly emerging general purpose technology, one that is already being eagerly adopted by developers, executives, and managers alike. While uncertainties remain around risks and long-term impacts, the transformative potential explains the excitement around unlocking the benefits of generative AI.

If you enjoyed this post please support our work by encouraging your friends and colleagues to subscribe to our newsletter: