ChatGPT in Research Theses: Study of the Reliability, Validity and Relevance of the Generated Texts

Abstract

The present study evaluates the quality of information generated by chatGPT, an AI tool often used by students during the realization of their research thesis. This evaluation focuses on three criteria, mentioned in the scientific literature consulted: fidelity (credibility of the source and that of the authors); validity (representativeness of the samples, robustness of the methods, accuracy of the data and objectivity of the authors); as well as the relevance (usefulness of the information generated). In order to obtain sufficiently relevant data on the quality of the information generated by the studied tool, the triangulation of methods is necessary. Three methods are indeed used: content analysis, data comparison and questionnaire survey. The analysis of research theses generated by chatGPT from unpublished topics, and the comparison of research theses made by students with those generated by chatGPT revealed that the information generated presents reliability, fidelity, and relevance issues; hence the need to verify them using more reliable references. These limits, the students interviewed are aware of them. Their responses to the electronic questionnaire, which was submitted to them, revealed positive representations of the studies toll: but betrayed a certain awareness of its limits. It is therefore up to teachers to make them aware of the risks of using this tool blindly, which can generate entirely erroneous responses. This unthoughtful recourse is justified by the informational incompetence of some of those students; which forces them to exploit the texts generated by chatGPT without checking their correction or adaptation to the context.