Piloting GenAI Tools for Qualitative Data Analysis

The Centre for Teaching, Learning, and Technology at the University of British Columbia has been exploring the use of generative AI tools for qualitative data analysis, specifically for open-ended survey responses. This work was prompted by a survey that received an unexpectedly high volume of open-ended responses, exceeding the team’s capacity for manual coding.

Through extensive exploration and testing, the team concluded that when a manually coded subset of the dataset is used to validate prompts and assess accuracy, genAI tools can be as effective as human coders at identifying themes and coding individual responses. This approach resulted in substantial time savings while maintaining analytical quality.

Using a large, existing qualitative dataset that had previously been manually coded, the Centre for Teaching, Learning, and Technology tested the efficacy of several major generative AI tools for qualitative analysis. The testing focused on two key tasks:

1. Generating themes consistent with those identified by human coders

2. Coding individual open-ended responses

Through this process, the team identified one publicly available tool that generated themes closely aligned with those identified by human coders, as well as one additional theme that had been missed during manual coding. To assess reliability at the response level, the dataset was broken into subsets, with subset size varying by question type. When applied in this way, the genAI tool coded individual responses at approximately the same level of inter-rater reliability as the human coding team.

Building on these findings, the team is now testing the same approach using a bespoke UBC-developed tool that offers expanded data protection and privacy options. Throughout this work, human expertise and oversight have remained central.

Core practices include disclosing to participants how generative AI will be used for analysis and the tool’s data policies, conducting human review to remove identifying information prior to analysis, validating AI-generated results against human-coded subsets of the data, and recognising that, as human-trained systems, genAI tools may exhibit bias and therefore require critical human verification.

In parallel, the team is working with colleagues across campus to share approaches to genAI-supported qualitative analysis and to establish emerging best practices for responsible, transparent, and effective use.

The primary impact of this work is a significant reduction in the time required to code large open-ended datasets. This allows us to include open-ended questions, and collect rich participant feedback, in surveys and contexts where we would not otherwise have the opportunity to do so. It also allows us to more quickly analyze open ended datasets as we can use early responses in the subset for testing.       

This was evidenced with the original application where, once the survey closed, one person was able to complete the AI-assisted analysis in one week. Had gen-AI not been used for the analysis, this would have taken several weeks and may have had to be limited to a randomly selected subset of the data.

• Access to previously human-coded qualitative datasets
• Careful prompt testing and validation using human-coded subsets
• Human oversight for bias detection and interpretation
• Participant transparency regarding AI use and data policies
• Institutional support for privacy-protective AI tooling