New Delhi: Researchers have found that AI models which are being trained to predict heart diseases and diabetes might have been built on datasets which have no verifiable source.
The study published in June in the journal BMC Medicine was led by researchers at the Queensland University of Technology and the Australian Centre for Health Services Innovation. Titled “Evidence of unreliable data and poor data provenance in clinical prediction model research and clinical practice” the study analysed two large datasets on Kaggle, an online platform for sharing datasets and machine-learning resources.
These two datasets, the stroke prediction dataset and the diabetes prediction dataset, had been used in 125 peer-reviewed studies, even though there was no information where that data came from, how it was collected, and whether it represented real patients.
“It was an enormous surprise to come across something like this. These datasets exhibit unusual patterns that raise serious questions about their authenticity and suitability for clinical research,” Alexander Gibson, lead author from the QUT School of Public Health and Social Work, said in a press release.
What was even more alarming was that three AI-based disease detection models built on these unverified datasets had already been used in clinical practice. One of these models was even cited in a medical device patent, and three of them were mentioned in nearly 86 review articles.
No verifiable proof
For the study, the researchers evaluated the dataset by looking at the TRIPOD+AI framework which is a guideline to make sure that clinical prediction models that use AI remain transparent and accurate. This framework checks who collected the data, where, when, under what conditions, whether the patients were real, whether ethical approvals had been gotten, what the sampling method was like. When scientists probed into the two datasets they found that there were no answers to any of those questions, and no proof that the dataset was real.
“Prediction models built on data of unknown provenance have no place in clinical decision-making. Without trustworthy data, the outputs are unreliable and risk misleading clinicians and harming patients,” Gibson added.
The authors of the study recommended that these two datasets in particular be removed from Kaggle to make sure there is no further misuse of the data. They added that such dataset providers must come up with stronger checks to make sure the data they provide is robust.
“We’re seeing fast‑churn research built on datasets that look scientific but lack the most basic transparency. Without stronger safeguards, unreliable models will continue to make their way into the literature, and potentially into practice,” said Gibson.