Assessing the Precision of AI-Generated Medical Answers: An Evaluation of LLaMA-3 Powered Meta AI
DOI:
https://doi.org/10.5195/ijms.2025.3887Keywords:
Artificial Intelligence, Medical Education, validation studies as topicAbstract
Background & Objective: Meta AI is being used frequently by medical students to clear the queries or to solve the questions, due to its easy availability. The purpose of this study is to assess the correctness of the Meta AI-generated answers to medical questions, and the reproducibility of the results.
Method: The study employs an Evaluation Research Design aimed to assess the quality and effectiveness of Meta AI. A total of 240 MCQs were included in the questionnaire, 30 MCQs from each subject. Out of these, 108 were case-based (Category: A), whereas 132 were fact-based (Category: B). Meta AI was re-queried with the previously failed questions 14 days later. Results were analyzed manually and accuracies were evaluated using IBM SPSS version 27.
Results: In initial analysis, Meta AI correctly answered 187 out of 240 questions (average accuracy = 77.9%). The most accurately responded category was Category-A with an average accuracy of 82.4%. The accuracy of Category-B was noted to be 74.2%. In re-scored analysis, Meta AI reproduced correct answers for only 12 out of 53 previously failed questions leading to an average reproducibility of 22.6%. The accuracy of Category A was 47.3% and that of Category B was 8.8%.
Conclusion: The integration of AI in the field of medicine is advancing rapidly, and models like Meta AI represent significant strides in making medical information more accessible and accurate. Despite these promising results, there are notable limitations, such as the scope of questions, the subjects covered, and potential selection biases.
Table 1. Overview of Total and Categorical Validation from Initial Analysis
Subject
Correct Answers
Wrong Answers
Category – A
n (total)
Category – B
n (total)
Accuracy %
Anatomy
20
10
1 (8)
9 (22)
66.6
Biochemistry
26
4
3 (16)
1 (14)
86.6
Community Medicine
20
10
5 (10)
5 (20)
66.6
Forensic Medicine
18
12
0 (0)
12 (30)
60.0
Microbiology
27
3
1 (16)
2 (14)
90.0
Pathology
29
1
1 (29)
0 (1)
96.6
Pharmacology
26
4
3 (15)
1 (15)
86.6
Physiology
21
9
5 (14)
4 (16)
70.0
Total =
187
53
19 (108)
34 (132)
77.9
Table 2. Overview of Total and Categorical Validation from Re-scored Analysis
Subject
Total Questions*
Correct Answers
Wrong Answers
Category – A
n (total)
Category – B
n (total)
Accuracy %
Anatomy
10
1
9
0 (1)
9 (9)
10.0
Biochemistry
4
1
3
2 (3)
1 (1)
25.0
Community Medicine
10
3
7
3 (5)
4 (5)
30.0
Forensic Medicine
12
1
11
0 (0)
11 (12)
8.3
Microbiology
3
2
1
0 (1)
1 (2)
66.6
Pathology
1
0
1
1 (1)
0 (0)
0.0
Pharmacology
4
3
1
0 (3)
1 (1)
75.0
Physiology
9
1
8
4 (5)
4 (4)
11.1
Total =
53
12
41
10 (19)
31 (34)
22.6
Legend: Total questions refer to the questions that were answered incorrectly in first (initial) attempt
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2025 Areeba Khan, Muhammad Talha Khan
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons Attribution 4.0 International License or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site; with the understanding that the above condition can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post online a prepublication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work. Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from the Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.
Enforcement of copyright
The IJMS takes the protection of copyright very seriously.
If the IJMS discovers that you have used its copyright materials in contravention of the license above, the IJMS may bring legal proceedings against you seeking reparation and an injunction to stop you using those materials. You could also be ordered to pay legal costs.
If you become aware of any use of the IJMS' copyright materials that contravenes or may contravene the license above, please report this by email to contact@ijms.org
Infringing material
If you become aware of any material on the website that you believe infringes your or any other person's copyright, please report this by email to contact@ijms.org
