New Delhi: The AI adoption has shot up exponentially in the last three years, with several models available to users. But to what degree and with what level of skill are people using them? A new report, Anthropic Education Report: The AI Fluency Index, found that the most common expression of AI fluency is ‘argumentative’—treating AI as a thought partner, rather than delegating work entirely.
Humans initiating meaningful and deep conversations with AI qualify as a more skilled use than simple chats.
One of its initial findings indicates that only 3.1 per cent of users are likely to question the codes, apps, documents or interactive tools produced by AI. And only 5.2 per cent of users in such cases can identify the missing context of the AI.
Studying behavioral pattern
The study focuses on 24 directly observable behaviours. Among them, only 11 have been selected, as these can be directly observed in chats.
The behaviors include ‘iterates and refines’, ‘clarifies before asking for help’, ‘provides example of what good looks like’, ‘specific format and structure needed’, ‘sets interaction mode’, ‘communicates tone and style preferences’, ‘identifies when AI might be missing context’, ‘defines audience for the output’, ‘questions when AI reasons do not hold up’, ‘consults AI on approach before execution’ and ‘checks facts and claims that matter’. The patterns are directly observable when humans interact with Claude, Anthropic’s AI.
To measure the AI fluency, the authors of the report used a tool developed by professors Rick Dakan and Joseph Feller in collaboration with the platform. The platform used a ‘privacy-preserving analysis’ tool to study 9,830 conversations, including several back-and-forth chats on Claude.ai. The study was done during a seven-day window in January.
The study found multiple behaviours in chats.
“This, finally, gave us the AI Fluency Index: a baseline measurement of how people collaborate with AI today, and a foundation for tracking how those behaviors evolve over time as models change,” states the study.
Also read: Meitei martial art Thang-Ta was banned under British rule. It survived as a performance form
Engagement pushes AI fluency to increase
AI fluency increases as the user works on the input, instead of accepting the first response and moving to a new task, according to the reartifiport. Nearly 85.7 per cent of conversations indicated patterns between exhibited iteration and refinement. The platform “built on previous exchanges to refine the user’s work, rather than accepting the first response and moving to a new task.”
“Conversations with iteration and refinement are 5.6x more likely to involve users questioning Claude’s reasoning, and 4x more likely to see them identify missing context,” states the study.
Nearly 12.3 per cent of people using AI to create codes, documents, or tools use AI differently. They state their goals and specify formats, give examples and iterate. However, users putting more effort into their prompts and directives do not add to AI fluency.
“These conversations are more likely to see users clarify their goal (+14.7 per cent), specify a format (+14.5 per cent), provide examples (+13.4 per cent), and iterate (+9.7 per cent) compared to non-artefact conversations. In other words, they’re doing more to direct AI at the outset of their work,” states the report.
(Edited by Saptak Datta)