J. David Giese’s Post

Are you considering pushing a 🙊LLM, 🙉LAM, 🙈LMM, or any large 💡 model through the FDA? This could save you trouble. We did a presub for a device that uses LLMs to detect contrast in the DICOM header info. How to FDA Approve LLMs. Step 1 Presubmission. https://hubs.li/Q02w3Qy90

OpenAI took a giant victory lap with o1, and it's advanced thinking abilities. One of their biggest claims was o1's supposedly superior diagnostic capabilities. However, after some research, I have reached the following conclusions- 1) OpenAI has been extremely negligent in their testing of the preview model, and has not adequately communicated it's limitations in their publications. They should do so immediately. 2) o1's estimation of the probability of having a disease given a phenotype profile is broken and inconsistent. For the same profile, it gives you different top-3 likely diseases. Another concerning observation: It gave a 70-20-10 probability split in 4/5 cases (with a different top 3 every time). This points to a severe limitation regarding the model's computations. 3) o1 also severely overestimated the chance of an extremely rare medical outcome, which could imply faulty calculations with prior and posterior probabilities. All of these lead me to conclude the following- 1. o1 is not ready for medical diagnosis. 2. To quote the brilliant Sergei (can't tag him, or anyone else, for some reason)- "OpanAI was overly cavalier in suggesting that its new o1 Strawberry model could be used for medical diagnostics. It’s not ready. OpenAI should apologize—they haven’t yet." 3. We need more transparent testing and evaluations in mission-critical fields like Medicine and Law. To read more about our research into the problems and possible solutions, read the following article- https://lnkd.in/dnZVq9cv

Unlike other suppliers in the field who have admitted to relying on Whisper, we are committed to delivering best-in-class accuracy and precision with our #MirAI models "Building reliable and precise medical speech recognition systems is no easy feat, especially in the healthcare sector where accuracy is paramount. Recent findings have highlighted a critical issue: leading models like Whisper have been shown to produce 'hallucinations' in transcriptions. In one instance, a machine learning engineer discovered hallucinations in nearly half of the over 100 hours of Whisper transcriptions he analysed. Another developer found them in almost every one of the 26,000 transcripts he created with Whisper." Read what researchers have uncovered in these articles on Slashdot and on Fortune: - https://lnkd.in/eW3STEEB - https://lnkd.in/eBSMDzCg At T-Pro, we don’t compromise on quality. We’ve developed our own proprietary ASR, powered by our #MirAI models, and engineered specifically to meet the high standards required in healthcare environments. To read more about MirAi: https://lnkd.in/eiWRkpZt When it comes to healthcare, trust matters. That’s why we focus on achieving the highest levels of reliability, ensuring our clients and partners can depend on us for the most accurate outcomes. #HealthcareInnovation #SpeechRecognition #ASR #MedicalTechnology #TrustInTech #MirAI

📣 Attention AI/ML developers, manufacturers and RA interested scrollers!!! New IMDRF document on "Good machine learning practice for medical device development - Guiding Principles" is now open for public consultation. We have been trying our very best to ensure that the most essential parts of creating a solid machine learning-enabled medical device are captured, but in case we missed anything, please let us know using the commenting template. Consultation closes on the 30th of August, so make sure to submit your comments before and enjoy the summer. 📅 🌞

Can GPT recognize handwritten (or bad-written) medical prescriptions? In this experiment, I used GPT-4o Vision and Retool to make a quick paper-to-digital converter tool. With common handwritten documents, it is surprisingly good at extracting text and even some formatting and structure. For medical prescriptions, it's still not enough (wait for the last example, I’d say, accurate) 😉

This work was created to automate the annotation of medical images by pre-segmentation, I think it is time to continue in this line. https://lnkd.in/eCRRuqz3

Ever wonder how AI is betting progressively better at understanding human language, including the nuances of medical language (still something of a work in progress)? Embeddings are the key! Basically a series of of floating-point numbers (think of long decimal values that are either positive or negative), they can translate complex medical terms into a numerical form that AI can process. Embeddings can capture relationships between words: Knowing "heart attack" is closer to "cardiomyopathy" than "common cold." Link to the interactive visual in the comments.

The use of collective intelligence methods to synthesize differential diagnoses, combining the responses of different LLMs, achieves three of the necessary steps toward advancing LLMs as a diagnostic support tool: demonstrating sufficiently high diagnostic accuracy, reducing the risk of misdiagnoses, and eliminating the dependence on a single commercial vendor. A promising way to augment doctor’s capacity to shorten the diagnostic gap and reduce the patient odissey!

EU MDR / IVDR News!! (Team-NB) Team-NB has published a position paper and questionnaire regarding the applicability of Artificial Intelligence (AI) in medical devices under MDR and IVDR regulations. Link: https://lnkd.in/dgnjHTnS  

I asked Chat GPT about NEXSTAR MEDICALS INT'L CO. LTD and see below the response I get.