Isabel Healthcare

THE VALIDATION PROCESS

This summarises the validation process that the Isabel Differential Diagnostic Tool (IDDT) has gone through to date and the forthcoming plans. Few decision support systems have undergone rigorous evaluation before being offered for general use. However, due to the considerable impact the IDDT could have we have chosen to put it through a robust validation process. In addition, it is important to remember that the IDDT is simply constructed by applying established software produced by Autonomy Corp to some of the most widely used medical textbooks. The resulting system effectively enables the user to interrogate a collection of textbooks in a rapid and intelligent way.

The full test protocols and results are available on request. The first two stages of testing were designed to show whether the concept of a diagnostic tool actually worked and came up with credible results. The third stage of testing, preceded by a simulated field trial, will not only test the IDDT in greater detail and across a broader spectrum of paediatric conditions but will also evaluate how the use of the system changes the clinician's management of patients.

Stage 1 Tests: completed in August 2000

The aim of these tests was to demonstrate whether the basic concept of a diagnostic tool worked. At this point the only textual content we had entered into the system were the specialities of Infectious Diseases and Rheumatology. Junior doctors and Consultants at one teaching hospital were asked for clinical case scenarios, ideally from personal experience, with the expected final diagnosis. The initial clinical features from the scenarios were then entered into the IDDT and the results were compared with a list of expected diagnoses. Out of the 99 case scenarios provided, the IDDT showed the expected diagnosis or diagnoses in 90 cases: an accuracy rate of 91%. The output list from the IDDT was programmed to limit the list of possible diagnoses to 15.

These results were presented to the Royal College of Paediatrics annual conference in April 2001 where Isabel was awarded first prize in its category.

Stage 2 Tests: carried out October -December 2000

The setting for stage 1 tests was clearly artificial so for stage 2 we aimed to evaluate the IDDT with real life clinical scenarios.

Junior doctors working in four A&E departments collected data from patients they assessed over a 3-month period in 2000. The data collected was: age group, presenting clinical features, the doctors' working diagnosis and the final diagnosis using the patient's discharge summary. Clinical features from 100 patients were entered into the IDDT and the output compared to the known final diagnosis. Out of these 100 cases, the IDDT showed the correct final diagnosis in 83 cases. In 13 of the 100 cases the final diagnosis was not a diagnostic category in the Isabel system (e.g. non-specific). When these 13 cases are removed the true accuracy rate was 95%. Again the output from the IDDT was limited to a maximum of 15 diagnoses.
These results have been submitted for publication in a peer reviewed medical journal.

Stage 3 Tests: Simulated Field Trial followed by Real Life Randomised Control Trials

Confident that the IDDT produces sensible results, the aims of the stage 3 tests are to test the efficacy of the doctor versus the combination of the doctor and IDDT in a real life setting. In order to gather a large volume of data we have designed a special version of the IDDT which allows a doctor participating in the trial to fill out the required form in a prescribed sequence entirely online. As an intermediate step to the full-scale trials we have carried out a simulated field trial which aimed not only to test the IDDT across a broad spectrum of conditions, but also to test the online forms themselves.

The simulated field trial was suggested to us by Jeremy Wyatt, one of the foremost world authorities on clinical decision support systems and advisor to the Department of Health; our protocol has been written following close discussions with him. The trial consists of 10 doctors from four different grades ranging from medical student through to consultant; each one tests the system with 12 case scenarios across a range of specialities as well as a range of difficulty ranging from the routine to the more unusual. The special trial version of the IDDT only allows the doctor to move forward through the cases in a set sequence. Most notably, he only has access to the output from the IDDT once he has entered his own differential diagnosis and management plan. The key outcomes that we will be trying to measure are:

1. the quality of the output from the IDDT
2. how it changed the doctor's management plan
3. how long it took him to evaluate each case.

Once we have fully analysed the results from the field trial and are satisfied that the software functions correctly we will then begin the full randomised control trials. These will be in a live setting at 10-12 large hospitals around the UK lasting over a six-month period to coincide with the new intake of junior doctors. With these trials the doctors will use the same special version of the IDDT as used for the simulated field trial but with real cases as they present at the hospital. The key aspects we will be trying to measure here will be the change in the diagnostic and patient management quality before and after using the IDDT. We shall also be attempting to measure the change in clinical outcomes and the time taken for each consultation. We anticipate that these randomised trials will run from September 2002 to March 2003.