Bixonimania, LLM controversy, and tips for using LLMs safely

In 2024, a research team in Sweden, led by Almira Osmanovic Thunström, uploaded two fake studies about a fabricated eye condition called “bixonimania” to the professional Preprints.org server.

The research team wanted to investigate whether large language models (LLMs) would recognise that the disease was fake or proliferate the information from the fake studies as fact. They deliberately added clear clues that the studies were fake, such as:

It did not take long before several LLMs picked up the misinformation and regurgitated it as reputable health information. The studies were even cited by several researchers, including in a study that was featured in a peer-reviewed journal.

The fake studies have since been deleted, and LLMs are now likely to correctly state that bixonimania is not a legitimate health condition.

But this still begs the question, how much can people really trust LLMs to provide reliable information—health-related or otherwise? The experiment has laid bare the ease in which LLMs can be manipulated to regurgitate misinformation.

In a world where healthcare can be expensive and difficult to access, it is understandable that many people are turning to LLMs for advice about their health.

However, LLMs are not a faultless technology. In some instances, they may even contribute to harming a person’s health.

For example, some people choose to seek mental health support from AI chatbots, which are powered by LLMs.

However, there are increasing reports of a phenomenon called “AI psychosis” in which AI chatbot use may trigger, amplify, or reshape psychotic experiences in vulnerable people. This may be due to AI chatbots reinforcing delusional beliefs, although, research into the phenomenon and its possible causes is ongoing.

There have even been some extreme instances in which AI psychosis has been linked to hospitalization and suicide.

However, more research into the topic is needed.

To conclude, the inherent and ongoing flaws in LLMs may highlight the profound importance of prioritizing human expertise and compassion—something the MNT editorial process emphasizes.

While LLMs might be a helpful starting point for someone who wants to learn more about various aspects of their health, the bixonimania experiment demonstrates that they can be a source of inaccurate information and cannot replace the expertise of a qualified healthcare professional.

“Consider LLMs/AI chatbots a starting point to understanding a medical situation, do not expect them to provide an accurate diagnosis or treatment plan.

“Never input personal information, such has your full name or address, insurance numbers, or details of your medical history. Always cross-check information that is provided.

“They can be helpful, especially if used to formulate questions or understand terminology.”

_{Quotes represent the opinions of our medical experts. All content is strictly informational and should not be considered medical advice.}