Research from Michael Lones, a professor at Heriot-Watt University’s School of Mathematical and Computer Sciences, warns that using generative AI in machine learning systems could lead to increased risks of cyber-attacks and data breaches.
Lones argues that while generative AI may offer cost and efficiency benefits, it could also expose organizations and the public to unintended harm, such as bias against underrepresented groups. He emphasizes the need for developers to balance capability improvements with associated risks.
The study, published in the journal Patterns, highlights how generative AI is progressively employed in designing and operating machine learning systems across various sectors. “Machine learning developers need to be aware of the risks of using Gen AI in machine learning,” Lones said.
Machine learning systems, which identify patterns in data for making predictions and decisions, are prevalent in daily life, including spam filters, product recommendations, and social media newsfeeds. High-stakes applications include drug trial assignments and processing insurance claims. Lones notes a significant increase in the push for incorporating generative AI, particularly large language models, into these systems.
However, the integration of generative AI entails substantial risks. Lones advises, “Avoid adding too much complexity in terms of how we use Gen AI in machine learning, particularly if you’re in a sector that has high stakes.” He identifies four uses of generative AI in machine learning: as components in machine learning pipelines, for designing pipelines, synthesizing training data, and analyzing outputs.
The risks accentuate when LLMs perform multiple tasks or function autonomously. Lones highlights that LLMs can make mistakes, poor decisions, and falsely generate information. These unpredictable errors complicate legal compliance due to their non-transparent nature.
“In areas like medicine or finance, there are laws about being able to show that the machine learning system is reliable,” Lones added. “As soon as you start using LLMs, that gets really hard, because they’re so opaque.” He stresses that the general public should recognize the limitations of generative AI, as companies may deploy these systems for cost-cutting, risking bias and unfairness.
The publication titled “Pitfalls and risks of generative AI in machine learning” appears in Patterns (2026), DOI: 10.1016/j.patter.2026.101534.
