Risks, and how to frame them
The technology is powerful and not safe by default. Hallucinations, privacy leaks, bias in the model and in yourself: they're all predictable. In this module: what they are, how to recognise them, and the four rules of thumb with which you become the safety layer.
What you'll learn: hallucination as feature, GDPR and the traffic-light model, two kinds of bias, four rules of thumb, and how to embed this locally (DPIA, governance, ESC AI Hub).
Lesson 5.1: Hallucinations: not a bug, a feature
Goal: never again be surprised that a good-sounding answer was nevertheless wrong.
A hallucination is, from the technical side, a statistically likely completion that happens not to be true. Not a bug, but inherent to the mechanism we discussed in modules 1 and 2.
Why this happens is no mystery: during training the model has seen what guideline citations look like. In the absence of exact memory, it fills the form with plausible-sounding content. It doesn't know that it doesn't know.
How to limit it (four measures)
- Provide the source yourself in the prompt (module 4.2). By far the most effective measure.
- Use tools with web search built in (ChatGPT web search; Claude in 2025 with web mode; Gemini Deep Research), they add verifiable citations.
- Ask explicitly for uncertainty + sources. Example: “Which parts of your answer are you least sure about, and why?”
- Verify every critical detail manually. Dose, percentage, year, class/level, paragraph number.
Questions for lesson 5.1
1. Which statement about hallucinations is most correct?
2. What's the most effective measure against hallucinating guideline citations?
Lesson 5.2: Privacy & GDPR: the traffic light
Goal: in 5 seconds know whether something is “green”, “amber” or “red” for the public chat.
The legal reality is that patient data in a public LLM is a data breach. Period. The practical reality is that many cardiologists still work in ChatGPT every day, often rightly so, with anonymised cases or their own raw points. The traffic-light model helps you distinguish in a few seconds:
- Entirely fictional case
- Your own raw dictation without identifiable data
- Writing a patient leaflet about a general condition
- Scientific question about a public guideline
- Course, teaching, presentation text
- Anonymised case with clinically rare combination (rare diagnosis + location + timestamp = possibly identifiable)
- Echo or ECG report of a real patient (without PII, but still with clinically unique pattern signature)
- Long pieces of your own record text, check there are no unique sentences in it
- Your own research data of patients (publication-ready or not)
- Complete discharge letter, admission report or record note with name, BSN, address or date of birth
- DICOM, ECG trace files, photos in which a patient is recognisable
- Patient data unique enough to make the patient re-identifiable
- Asking follow-up diagnosis questions as if about a specific patient with PII attached
- “Only I use it, so it's allowed”, false assumption
The “own environment” exception
Many hospitals in 2025–2026 have Microsoft Copilot for M365 (business), Azure OpenAI, or a specific Claude Enterprise environment with DPIA and processor agreement. There the rules are often wider (with local logging, no training on your data, possibly within EU). Ask ICT/CIO or a Data Protection Officer what's available and which data is allowed there. The course can't decide that for your workplace.
Questions for lesson 5.2
3. Which of these actions is “red” in a public chat (ChatGPT.com / Claude.ai)?
4. A case without name/BSN but with “67-year-old woman, living in [small village], admitted on [exact date] with [rare diagnosis]”, what colour in the traffic light?
Lesson 5.3: Bias in the model
Goal: know where the data is skewed, and therefore where the answer is skewed.
The training data of LLMs reflects the society that produced it, including its skewed distributions. For the cardiologist, three concrete areas to be alert to:
- White-male over-representation. Much cardiology literature up to the 2010s is predominantly based on white men. Advice for women and non-European populations can fit less well.
- AMI symptoms in women historically under-reported. The “typical” presentation the model gives leans heavily on the male presentation. Atypical complaints, fatigue, nausea, shoulder pain, get less attention in summaries than they deserve.
- English-language medical literature dominates. Rare local guideline particularities (society-specific working agreements, local policy) the model often doesn't know.
Practical habit
With every clinical recommendation from an LLM for a woman or a patient with non-European background: explicitly ask “how would this advice be different for a female patient” or “which groups are under-represented in the sources this advice leans on?”. Not as a dutiful tick, but as a correction question that sometimes yields real adjustments.
Lesson 5.4: Bias in the user: automation bias
Goal: recognise yourself in a well-documented pattern.
Automation bias is the tendency to accept answers that sound structured, fluent and self-assured faster than answers from a colleague. Three observations from the field:
- We accept AI output faster than a colleague's output, while the opposite should hold.
- The busier you are, the less you verify, precisely when errors cost the most.
- “The AI said so” feels like cover, while you remain ultimately responsible.
The cardiology context makes this acute: confirming a DD in the ED under time pressure seems the same action as “consult colleague”, but is fundamentally something else. An LLM has no clinical context, no patient feel, and no responsibility.
Three self-checks
- Are you really reading the AI answer, or just scanning it for “feels logical”?
- Would you accept the answer as quickly if a resident had said it?
- What is not in the answer that a colleague's answer would include?
Questions for lesson 5.4
5. What is automation bias?
Lesson 5.5: Four rules of thumb to deploy it safely
1. Provide the context yourself
Paste the guideline, the paper or the record fragment in the prompt. Then it works with your source, not with vague memory.
2. No patient data in public tools
Use only hospital-approved environments for real data. Otherwise: anonymise or construct.
3. Verify what matters
Not every word. But every number, every dose, every guideline statement, every literature reference. No exceptions.
4. You remain ultimately responsible
The AI is a good resident. Final responsibility for the patient is yours. Legally and morally.
Questions for lesson 5.5
6. What's a workable formulation of “verify what matters”?
Lesson 5.6: Local governance and what professional societies do
Goal: know where to anchor yourself as a department wanting to use AI more broadly.
Good news: you're not alone. In 2025–2026 substantial work has been done by professional societies and research groups you can lean on:
- ESC Artificial Intelligence Hub, central place for AI resources, masterclasses and the Focus on Digital Tools and AI in Cardiology track.
- ESC Digital Cardiology and AI Committee (DCAI), building a roadmap for safely integrating AI into cardiology. Data quality, evaluation frameworks, stakeholder engagement.
- EACVI 2025+, own AI track within multimodal imaging (echo, CMR, CT, nuclear). Hands-on training in the Imaging Training Hub.
- NVVC and your own hospital DPIAs, actively ask what's locally available for “own environment” LLMs.
What you as an individual cardiologist or trainee can do
- Find out which tools your employer has already approved (often: Microsoft Copilot M365, sometimes Azure OpenAI, sometimes Claude Enterprise). Work there, not in a personal account.
- Ask whether there is a DPIA for LLM use on the department. If not, start one or report it to the Data Protection Officer.
- Keep a personal register of use cases for which you deploy an LLM. Helps with audits and collegial discussions.
- Share errors and hallucinations you encounter with the department. Culture is preventive: a “this went wrong” moment prevents repetition by colleagues.
Lesson 5.7: Final exercise: hallucination hunt
Goal: identify a hallucination in a real chat and think how you should have asked the question.
Live exercise: find the hallucination
Step 1. Ask this question (deliberately tempts the model into a guideline citation):
Step 2. Examine the answer critically. Chapter number? Table? Class? “Direct quote”? If possible, open the real ESC 2023 endocarditis guideline (publicly available on the ESC site) and compare. Step 3. Then ask this safer follow-up in the same chat:
The difference between step 1 and step 3, form A “asking source” vs form B “providing source”, is the entire essence of this module.
Take-home from module 5: and from the course
Three things to take into practice, not just for this module but for the entire course:
1. One line, from which everything follows logically
An LLM predicts the most likely next chunk of text. It doesn't think, knows nothing, but imitates language convincingly. From which follows: provide source, question at the bottom, and verify what matters.
2. At minimum an excellent resident
Letters, summaries, code, first DD, leaflets, teaching, there it already adds value today. The pace of progress is high: try again periodically, including use cases that didn't work last year.
3. You are the safety layer
Provide context. No patient data in public tools. Verify what matters. You bear ultimate responsibility, always. No AI disclaimer protects you; your own clinical judgment does.
Good luck applying it. Questions? Reflection moments with colleagues around an “AI mistake of the week” often do more than another new course.