Researchers: ‘Overreliance on AI output undermines critical thinking’

Those who continuously rely on the outcomes of generative artificial intelligence (GenAI), risk to reduce the cognitive effort that is required for important skills, such as critical thinking.

This reduction in cognitive effort is more pronounced among users who have a high confidence in the capabilities of AI.

That’s the main conclusion of scientists from Microsoft and the Carnegie Mellon University. Researchers asked 319 knowledge workers who use GenAI tools at work at least once a week to tell about their experiences with GenAI. Their findings have been documented in the report ‘The Impact of Generative AI on Critical Thinking’.

According to the researchers, those who have great confidence in the skills and output results of AI, seem to have the idea that AI can think critically. This leads to less critical thinking on their own part. On the other hand, people who have more confidence in their own skills, tend to distrust AI output and do not automatically accept the outcome as true.

“The data shows a shift in cognitive effort as knowledge workers increasingly move from task execution to oversight when using GenAI,” researchers say. So instead of directly performing tasks, knowledge workers shift to overseeing AI outputs. Verifying information and integrating AI responses become more important.

In general, participants were aware of the issues of hallucination in GenAI. Especially those who work in areas where faulty AI output could be life-threatening, like in the medical world, tend to double-check the outcomes.

The scientists have identified several motivators for and barriers to critical thinking. Ensuring work quality, avoiding negative consequences of AI errors, and developing personal skills are considered motivators for critical thinking. Overreliance on AI output, lack of recognition of potential errors in AI output, time constraints, and difficulty in improving AI responses have been identified as barriers to critical thinking.

“While GenAI can improve worker efficiency, it can inhibit critical engagement with work and can potentially lead to long-term overreliance on the tool and diminished skill for independent problem-solving. Higher confidence in GenAI’s ability to perform a task is related to less critical thinking effort,” Microsoft and the Carnegie Mellon University conclude.

The researchers say AI companies should be aware of this potential dependency and encourage workers to have a more critical attitude toward AI output.