Evaluations

Independent safety evaluations of medical AI

We test how medical AI systems behave on controlled clinical cases, with physician reference assessments and particular attention to errors that could change what happens to a patient.

What We Evaluate

How medical AI perceives, integrates, interprets, and acts on clinical information across text, audio, and video.

  • Clinical Perception

    Does the system notice relevant visual findings, abnormal speech, breathing, movement, behavior and environmental clues? A cue that is visible for a moment may be lost to frame sampling or resolution before any reasoning happens, so input conditions are recorded with each result.

  • Multimodal Synthesis

    Does it combine transcripts, audio, video and structured data coherently, especially when the sources disagree? We test which source it relies on and whether it notices the contradiction.

  • Clinical Interpretation

    Does it understand the clinical significance of what it observed, including the differential, the uncertainty and any time-critical risk? A finding can be noticed and still be misread.

  • Clinical Decisions

    Does it recommend appropriate urgency, escalation, reassessment, management and disposition? A plausible diagnosis paired with an unsafe discharge is a different failure from a missed finding.

How results are assessed

Each case has criteria written for it in advance, and the clinically significant failures are defined before testing, so results cannot be reshaped afterwards. Outputs are compared with assessments by board-certified physicians, which serve as references rather than ground truth: where reviewers disagree in a way that matters, the disagreement is reported and adjudicated, not averaged away. Errors are weighted by their clinical consequence, so an unsafe discharge does not count the same as a minor omission. Inputs, system versions and settings are recorded so a finding can be reproduced, and each report states what was tested and what was not.

What can be tested

Multimodal clinical models, triage and decision-support systems, commercial products and product demonstrations, and research models applied to clinical tasks. A system can be evaluated through the interface its users see, through an API, or with deeper access from its developer. What an evaluation can establish depends on that access, and the report says which was used.

Controlled comparisons

  • The same case presented as text, audio, video or all three, to see which input changes the answer.
  • Matched variants with a finding present or absent, more or less obvious, or degraded.
  • Atypical, borderline and adversarial cases, including misleading notes and framing.
  • Repeated runs, to measure consistency when outputs vary.

Cases often come from Greyoak’s own physician-benchmarked collections, held out from any training material, or are written for the system being tested.

What you receive

Case-level findings with the input, the model output, the relevant timestamps, the observed failure and its potential clinical severity, and how often it recurred. Depending on the engagement, this is delivered as an engineering report focused on reproducible problems, a comparison with physician reference assessments, or a confidential independent assessment for a developer, investor or counsel.

Independence

Commissioning an evaluation does not determine its findings. Scope and questions are agreed in advance, and we report what we find, whether or not it is favorable. Confidentiality is set out in the terms of each engagement.

A Greyoak evaluation is not a regulatory clearance or certification, and it does not establish that a system is safe for clinical use.

Discuss an evaluation