News

Major Dev Survey Tackles AI, Ranks AWS CodeWhisperer Highly

• By David Ramel
• 06/26/2023

Stack Overflow's huge developer survey this year tackled AI, revealing how coders are using hot new generative AI tech and giving a high ranking to the AWS CodeWhisperer tool.

To be sure, the Amazon coding assistant ranked far behind GitHub Copilot but still came in at third place in the list of AI developer tools, leading the cloud-hosted offerings. That's not bad for a tool that only became generally available last month after a year-long preview.

One of -- it not the -- pre-eminent developer surveys in the industry, Stack Overflow's annual report this year polled some 90,000 people from all over the world to learn their top picks for tools and programming languages they are using along with just about every other professional and personal detail that could be considered.

With the recent debut of ChatGPT, GitHub Copilot, GPT-4 and all the rest, SO naturally added an AI section for the 2023 report.

AI highlights of the report as presented by SO include:

• 70 percent of respondents are already using or plan to use AI tools in their development process: 44 percent use AI tools now and 26 percent plan to soon.
• 77 percent feel favorable towards using AI tools as part of their development workflow -- but we do see some differences between groups.
• 33 percent see increasing productivity as the most important benefit of using AI tools as part of a development workflow.
• 42 percent trust the accuracy of the output of AI tools used in their development workflow.

One perhaps surprising finding is that while developers are keenly interested in AI tech, they aren't really using that many AI different tools in their work.

"It's early days in the hype cycle for these newer AI technologies," SO said in a June 14 blog post. "We expect that a little more time may need to pass before we see developers using more AI tools more broadly. Another factor that may be contributing to the slow adoption of AI tools into the development process is that professional developers don't highly trust the accuracy of AI tools. Of those using or planning to use AI tools, only 3 percent highly trust the accuracy they provide, and twice as many (6 percent) highly distrust them. While the majority (39 percent) are somewhat trusting of AI tools, this response tells us that AI tools still need to prove their usefulness."

Among those tools, as mentioned, GitHub Copilot was the far-and-away leader, mentioned by nearly 55 percent of all respondents, followed by Tabnine (nearly 13 percent) and AWS CodeWhisperer (a little over 5 percent). After those three, no other tool outranked the "Other" category.

As noted, the survey did show that 70 percent of respondents were using or planning to use AI tools, adding an interesting data point to compare with a recent GitHub report that found a "staggering" 92 percent of U.S. developers were using AI tools.

As far as those "trust" and "usefulness" concerns mentioned by SO above, a couple graphics show the big picture:

The latter graphic is somewhat comparable to one in that aforementioned GitHub report:

Looking forward, SO also asked developers how they thought AI tools would evolve their workflow over the next year, finding that 77 percent of respondents felt that the next year will look somewhat or very different for writing code with AI -- with 75 percent feeling the same about debugging code with AI. Updating the tools and evolving the learning curve will likely benefit those currently using AI tools or planning to soon, SO said.

The sprawling survey goes on to cover much more than AI, of course, delving into areas such as developer profiles, technology used (and admired/desired), community participation and much more, but AI was clearly the shining new star in this report.

"This year, we went deep into AI/ML to capture how developers are thinking about it and using it in their workflows," SO said. "Stack Overflow is investing heavily in enhancing the developer experience across our products, using AI and other technology, to get people to solutions faster. Stack Overflow Labs is where we're sharing all we're doing -- check it out for a deep dive on AI/ML insights as well as see what we're experimenting with so far."