Computer-based Screening for Diabetic Retinopathy

Phase 0 Observational Study of a Computer-based Screening Algorithm for Detection of Diabetic Retinopathy

The study is an Observational, Phase 0 designed to establish that the risk for diabetic retinopathy assigned by the RiskAnalyzer improves the reading accuracy and consistency of any reader and it decreases the inter-reader variability.

Objectives: Objectives one, two, and three are arranged chronologically and in an increasing level of complexity as a three tiered approach to support the primary and secondary endpoint of the trial. Objective one is to test fully each system components of the study limited to a single site. Objective two is to evaluate the efficacy of the RiskAnalyzer to assign the risk of Diabetic Retinopathy in comparison to the gold standard. Objective three is to demonstrate that the reader's accuracy in grading images is improved when risk levels assigned by the RiskAnalyzer are made available to the reader while performing the grading of the images which is the primary endpoint of the trial Methods and Research Design: A network of clinical study sites will be established to meet the required number of cases needed as calculated by statistical analysis. Male and Female Subjects between the ages of 18-65 who are either pre-diabetic or diabetics will be eligible for participation in this study. Subjects will be recruited, consented, photographed and their images graded by two trained readers and analyzed by the RiskAnalyzer . The risk levels that are obtained from the RiskAnalyzer will be compared to the current gold standard practice, manual grading of each case by a reader. Data collected during this clinical trial will be reported to the referring physician in the form of a retinal screening report completed and signed by a licensed Ophthalmic professional and delivered to their attending physician. Risk levels for diabetic retinopathy obtained by use of the RiskAnalyzer will not be given to the attending physician under any circumstances in order to preserve standard of care for the patient.

The sensitivity, specificity, receiver operating characteristic (ROC), and data flow process of the RiskAnalyzer, retinal image reading system will be analyzed and based on the current gold standard of a human reader. This study is a three-aims study, 24 month in length, prospective, case-only study of the performance of the RiskAnalyzer. The risk levels obtained from the RiskAnalyzer are not made visible, i.e. are not unmasked to either of the two readers. In year two, the risk levels obtained from the RiskAnalyzer for half of the studies are unmasked to the two readers while grading the image.

Access to all study data and processes follows a role-based design. The clinical staff will have access only to clinical data but not the technical data. The technical team will have access to the technical data only but not the clinical data. The study coordinator will have access to all data. The use of computers will adhere to the Guidance for Industry Computerized Systems Used in Clinical Investigations and applicable sections of 21 CFRs part 11.

Study Overview

• Status: Completed
• Conditions: Diabetic Retinopathy

Detailed Description

Sensitivity of retinal screening for risk stratification of DR using digital fundus photographs with and without the assistance of the RiskAnalyzer

Sensitivity and specificity of retinal screening for risk stratification of DR using digital fundus photographs and the RiskAnalyzer

Clinical Trial Methods and Research Design: A total of 10,000 subjects will be recruited to participate in this study. These subjects will be enrolled through two parallel collaborators, Extension for Community Healthcare Outcomes (Project ECHO) and CommuniCare Health Centers. Each clinic will obtain approximately 50% of the cases per year or an average of 50 cases per week. Subjects will be identified from clinics, CommuniCare Health Centers of San Antonio, TX and Project ECHO, New Mexico. CommuniCare has two associated sites, the Barrio Family Health Center in Southeast San Antonio, TX and the Dr. Frank Bryant Health Center also in San Antonio. CommuniCare serves approximately 45,000 subjects a year of which it is estimated that 5000 are requiring diabetic care. Project ECHO in conjunction with the University of New Mexico collaborates with over 20 rural clinics in New Mexico to provide service to underserved areas in New Mexico. A recruitment flyer and consent forms will be provided to each site. Clinic staff will recruit subject for the study during scheduled visits. Subject will be given a flyer containing information about the study and the informed consent document. Subjects who are interested in being enrolled in the study will be referred to a photographer within the site for Retinal imaging. Subjects will not receive a payment or stipend for participation in this study.

There are no screening requirements for subjects to participate in this trial. Subjects must be diagnosed as a person with pre-diabetes or diabetes by a medical professional. There is no requirement or solicitation for information regarding prior or concomitant therapy. There are no criteria set forth for subjects to exit or be discontinued from the study once the retinal screening is performed. Subjects will not be withdrawn from the study once retinal screening has occurred. Once the subject completes retinal screening, the period of subject participation is concluded. Images taken from subjects may not be used in the study due to inadequate image quality. The number of images found to be inadequate will be tracked and reported at the completion of the study. Retinal screening marks both the beginning and the end of subject participation in this study.

For each eye, a minimum of two posterior pole images will be collected. One image will be centered on the fovea and the second on the optic disc. A third image will be collected of the anterior segment (front of the eye). Subjects will not be pharmacologically dilated. Imaging will rely on natural dilation due to adaptation to darkness in the imaging room. More than two images per eye will be collected only when the imager (photographer) identifies any given image of insufficient quality for reading. Poor quality images are typically due to physiological reasons, such as blinking or small pupils and will not be used as part of this study. The retinal photographer will attempt up to 3 times to acquire images of a subject's eyes in order to complete the imaging protocol. The study coordinator will conduct weekly audits of the imaging procedure in order to ensure and document compliance with the study protocol.

The Class II Medical Device that will be used for this study will be a non mydriatic or equivalent retinal camera. The retinal camera is intended to be used for taking digital images of the retina of the human eye without the use of pharmacological dilation. The retinal camera has the following photography modes: RGB color. A Digital Camera is mounted to the retinal camera to perform retinal imaging.

A picture archival and communication system (PACS) is used to receive DICOM images, scheduling information and textual reports, organize and store them in an internal format, and to make that information available across a network via HIPAA and FDA-cleared secure user interfaces. The PACS is comprised of a capture component, a central server component, and a viewer component that is used on general purpose computing hardware. Subject information, procedure information, and image acquisition data is captured, sent to a server or alternatively, it is accepted directly from a DICOM compliant device. After receiving image objects and associated data, the server registers the incoming data and images with the archive including any associations with prior cases. The cases are presented to users with the correct permissions from a work list as controlled by a managing user. The image data is transmitted and rendered on the user's workstation using a standard web browser. Once the server registers a report, the report is available to the referring physician or client service electronically or via fax.

Immediately after the images and Subject demographics are collected, the photographer will upload the data as a case to the PACS via a VPN connection established between the imaging workstation and PACS. If the VPN connection cannot be established at the time of imaging, the upload will be done at a later time, e.g. end of the clinic day. The data will reside in the imaging workstation for up to five days in order to assure upload. The clinical coordinator will conduct weekly audits of upload completion. Once in PACS, cases are read by readers using a built in reading template that generates a report with their findings. The study will not disrupt standard of care to the study subjects. Each site will follow its own standard procedures and care. Findings will be reported to the clinic which referred the subject for screening. Clinics will be responsible for follow-up care including any necessary intervention or therapy.

Performance of the RiskAnalyzer: The performance of the RiskAnalyzer will be calculated by applying the minimally acceptable values for sensitivity (Seo) and specificity (Spo) values specified in advance. For purposes of this study, the specified null hypotheses values are Seo=0.85 and Spo=0.70. These values are based on data obtained from ophthalmic exams. The efficacy of trained readers using retinal photography has been demonstrated to provide the most sensitive screening and monitoring test for sight-threatening retinopathy, with sensitivities > 80%. Abramoff, et al. discuss the limits of sensitivity that can be achieved by any system given the fallibility of ground truth based on multiple human readers. He suggests that 0.85 is the limit of sensitivity that one can achieve. The specificity is based on the efficiency that we seek in order to make this implementation commercially viable. We have designed our study to achieve 0.90 statistical power and 0.99 significance level. If confidence limits are based on asymptotic normal distribution theory, then sample sizes can be based on the asymptotic variance formulae, as given by Pepe.

Objective One: Objective one is to test fully the study data flow logistics and information transfer interfaces. The testing will be done entirely at one recruitment, enrollment and imaging site. The duration of time to complete this objective is three months.

Materials and Research Design for Objective One: Subjects will be recruited from single site. All images will be routed to two readers for grading. If the two readers assign equivalent grades to the image, a generated report will be given to the clinic to distribute the referring physician. For cases in which the readers assign different grades for the same image, the image will be routed to an adjudicator for resolution. The adjudicator will have access to the reports generated by each reader, but not the risk levels from the RiskAnalyzer. A report from the adjudicator will be generated and reported to the clinic that will be responsible for distributing the report to the Subject and for any further follow-up. Images will also be de-identified and routed to the RiskAnalyzer. The RiskAnalyzer will perform its analysis of the image and generate a risk stratification of either "Low risk" or "High risk" for diabetic retinopathy. A report containing the risk levels assigned by the RiskAnalyzer will not be distributed to the referring physician, clinic or adjudicator but used only for statistical analysis in support of the endpoints of the trial.

Outcomes for Objective One: Data outcomes will demonstrate the ability to recruit and enroll subject into the study at a single site, perform retinal imaging, upload the images to a PACS, have the same images graded by two different readers, when necessary routed and graded by an adjudicator and the ability to generate a final report. The reading process followed by the readers is based on the International Clinical Diabetic Retinopathy Disease Severity Scale. Additionally data will demonstrate that images are successfully routed to the RiskAnalyzer for processing.

Objective Two: Objective two is to evaluate the efficacy of the RiskAnalyzer to identify the risk of Diabetic Retinopathy in comparison to the gold standard. The duration of time to complete this objective is nine months.

Methods and Research Design for Objective Two: Subjects are recruited from multiple CommuniCare and Project ECHO sites. All images are graded by two readers and an adjudicator should the grading by the first and second reader not be in agreement. The reading process followed by the readers is based on the International Clinical Diabetic Retinopathy Disease Severity Scale. A consensus report generated by the readers or the adjudicator indicating presence and severity of diabetic retinopathy will be given to the Subject through their referring physician and referred to a specialist if necessary by the referring physician. The same images graded by the readers will also be routed to the RiskAnalyzer for processing which will generate a risk level of either "Low risk" or "High risk" for diabetic retinopathy. As the images are routed to the RiskAnalyzer, the images will be de-identified and sequestered until evaluated by the RiskAnalyzer. Risk levels obtained by use of the RiskAnalyzer will not be given to the Subject under any circumstances. The referring physician is also masked from the risk levels assigned by the RiskAnalyzer. Referring physicians are not co-located at VisionQuest Biomedical and therefore will not have any access to the computers or data storage area for the data.

Outcomes for Objective Two: Statistical analysis will indicate the accuracy level of the RiskAnalyzer as compared to the gold standard. Statistical analysis will use sensitivity, specificity, AUC, and a level of agreement using Cohen's kappa coefficients.

Objective Three: Objective three will address the primary endpoint of the trial which is to demonstrate that the reader's accuracy in grading images is enhanced when risk levels from the algorithm are made available to the reader while performing the grading of the images. The duration of time to complete this objective is 12 months.

Methods and Research Design for Objective Three: The RiskAnalyzer will be used as a first reader. There will continue to be two human readers and an adjudicator. Ground Truth will be established by either the agreement of reader one with reader two or by an adjudicator. All images will be routed to the RiskAnalyzer which will generate a risk level of either "Low risk" or "High risk" for diabetic retinopathy. The cases will be divided into two sets and routed to both readers. One set of images will be routed to both readers with the algorithm assigned risk levels while the other set of images will be routed to both readers without the algorithm assigned risk level. An equivalent number of 'Low risk" and "High risk" cases will be selected for each set of images. Ground Truth is established by the two readers assigning the same grade to the same image. When different grades are assigned to the same image by two different readers, the cases will be routed to an adjudicator for grading and reporting. The adjudication will become ground truth. The cases that will be adjudicated will only be those that are different categories of risk stages. Adjudication will be based on the clinical significance of the disagreement between the readers as follows: cases within severity scales 0 and 1 will be classified as "Low Risk" cases, while cases within the severity scales 2 and above will be classified as "High Risk" cases. Cases needing adjudication will be those differing in risk and those within the High Risk Category that differ between scales 2 and 3 (mild and severe), and 2 and 4 (mild and proliferative Diabetic Retinopathy -PDR).

Outcomes for Objective Three: The outcomes from objective three will establish that the RiskAnalyzer improves the reading accuracy and consistency between readers and decreases bias between human readers. Statistical Analysis of level of agreement will be performed on the risk assigned to the images by the readers when having and not having the risk levels assigned by the algorithm using Cohen's kappa coefficients.

Expected outcomes: Telemedicine retinal screening in conjunction with risk levels from the RiskAnalyzer improve the accuracy of readers when grading images. The use of risk levels assigned by the RiskAnalyzer in assisting the readers will decrease the amount of time readers dedicate to the grading of each case. This may be a model to support the increase of access to retinal screening, support compliance with retinal screening recommendations and facilitate a more efficient use of the retinal specialist to evaluate and treat disease leading to a more cost efficient method of care.

• Study Type: Observational
• Enrollment (Actual): 19199

Contacts and Locations

Study Locations

• United States
  • New Mexico
    • Albuquerque, New Mexico, United States, 87102
      • Project ECHO
  • Texas
    • San Antonio, Texas, United States, 78207
      • CommuniCare Health Centers of San Antonio

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years to 65 years (Adult, Older Adult)
• Accepts Healthy Volunteers: No
• Genders Eligible for Study: All

• Sampling Method: Probability Sample

Study Population

Primary care clinics in the Southwest serving a Diabetic Community

Description

Inclusion Criteria:

• Patients over the age of 18
• Patients diagnosed as with either PreDiabetes or Diabetes type 1 or type 2

Exclusion Criteria:

• People not diagnosed with either PreDiabetes or Diabetes type 1 or type 2

Study Plan

How is the study designed?

Design Details

• Observational Models: Case-Only
• Time Perspectives: Prospective

Number of groups / cohorts

1

Cohorts and Interventions

Group / Cohort
Retinal Screening; Diabetes 1 and 2; Patients diagnosed as either having Pre-Diabetes, Diabetes 1 or Diabetes 2

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Sensitivity and specificity of retinal screening for risk stratification of DR using digital fundus photographs with and without the assistance of the RiskAnalyzer.	Using the gold standard, the sensitivity and specificity are calculated as: Sens=TP/(TP+FN) Spec=TN/(TN+FP) Where TP is the number of true positives (DR subjects correctly classified), FN is the number of false negatives (DR subjects incorrectly classified as normal), TN is the number of true negatives (Normal subjects correctly classified), and FP is the number of false positives (Normal subjects incorrectly classified as abnormals).	24 months

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Sensitivity and specificity of retinal screening for risk stratification of DR using digital fundus photographs and the RiskAnalyzer	Using the gold standard, the sensitivity and specificity are calculated as: Sens=TP/(TP+FN) Spec=TN/(TN+FP) Where TP is the number of true positives (DR subjects correctly classified), FN is the number of false negatives (DR subjects incorrectly classified as normal), TN is the number of true negatives (Normal subjects correctly classified), and FP is the number of false positives (Normal subjects incorrectly classified as abnormals).	24 months

Collaborators and Investigators

• Sponsor: VisionQuest Biomedical LLC
• Collaborators: National Eye Institute (NEI); Project Extension for Community Healthcare Outcomes (ECHO); Communicare Health Centers of San Antonio; Retinal Institute of South Texas
• Investigators: Study Director: Gilberto Zamora, PhD, Director of Clinical Operations

Publications and helpful links

General Publications

• Griffith SP, Freeman WL, Shaw CJ, Mitchell WH, Olden CR, Figgs LD, Kinyoun JL, Underwood DL, Will JC. Screening for diabetic retinopathy in a clinical setting: a comparison of direct ophthalmoscopy by primary care physicians with fundus photography. J Fam Pract. 1993 Jul;37(1):49-56.
• Williams GA, Scott IU, Haller JA, Maguire AM, Marcus D, McDonald HR. Single-field fundus photography for diabetic retinopathy screening: a report by the American Academy of Ophthalmology. Ophthalmology. 2004 May;111(5):1055-62. doi: 10.1016/j.ophtha.2004.02.004.
• Lin DY, Blumenkranz MS, Brothers RJ, Grosvenor DM. The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: a comparison with ophthalmoscopy and standardized mydriatic color photography. Am J Ophthalmol. 2002 Aug;134(2):204-13. doi: 10.1016/s0002-9394(02)01522-2.
• Boucher MC, Gresset JA, Angioi K, Olivier S. Effectiveness and safety of screening for diabetic retinopathy with two nonmydriatic digital images compared with the seven standard stereoscopic photographic fields. Can J Ophthalmol. 2003 Dec;38(7):557-68. doi: 10.1016/s0008-4182(03)80109-6.
• Jones CD, Greenwood RH, Misra A, Bachmann MO. Incidence and progression of diabetic retinopathy during 17 years of a population-based screening program in England. Diabetes Care. 2012 Mar;35(3):592-6. doi: 10.2337/dc11-0943. Epub 2012 Jan 25. Erratum In: Diabetes Care. 2014 Mar;37(3):887.
• Agurto C, Murray V, Barriga E, Murillo S, Pattichis M, Davis H, Russell S, Abramoff M, Soliz P. Multiscale AM-FM methods for diabetic retinopathy lesion detection. IEEE Trans Med Imaging. 2010 Feb;29(2):502-12. doi: 10.1109/TMI.2009.2037146.
• Agurto C, Barriga ES, Murray V, Nemeth S, Crammer R, Bauman W, Zamora G, Pattichis MS, Soliz P. Automatic detection of diabetic retinopathy and age-related macular degeneration in digital fundus images. Invest Ophthalmol Vis Sci. 2011 Jul 29;52(8):5862-71. doi: 10.1167/iovs.10-7075.
• Larsen M, Godt J, Larsen N, Lund-Andersen H, Sjolie AK, Agardh E, Kalm H, Grunkin M, Owens DR. Automated detection of fundus photographic red lesions in diabetic retinopathy. Invest Ophthalmol Vis Sci. 2003 Feb;44(2):761-6. doi: 10.1167/iovs.02-0418.
• Niemeijer M, van Ginneken B, Russell SR, Suttorp-Schulten MS, Abramoff MD. Automated detection and differentiation of drusen, exudates, and cotton-wool spots in digital color fundus photographs for diabetic retinopathy diagnosis. Invest Ophthalmol Vis Sci. 2007 May;48(5):2260-7. doi: 10.1167/iovs.06-0996.
• Niemeijer M, van Ginneken B, Staal J, Suttorp-Schulten MS, Abramoff MD. Automatic detection of red lesions in digital color fundus photographs. IEEE Trans Med Imaging. 2005 May;24(5):584-92. doi: 10.1109/TMI.2005.843738.
• Osareh A, Mirmehdi M, Thomas B, Markham R. Automated identification of diabetic retinal exudates in digital colour images. Br J Ophthalmol. 2003 Oct;87(10):1220-3. doi: 10.1136/bjo.87.10.1220.
• Scotland GS, McNamee P, Philip S, Fleming AD, Goatman KA, Prescott GJ, Fonseca S, Sharp PF, Olson JA. Cost-effectiveness of implementing automated grading within the national screening programme for diabetic retinopathy in Scotland. Br J Ophthalmol. 2007 Nov;91(11):1518-23. doi: 10.1136/bjo.2007.120972. Epub 2007 Jun 21.
• Fleming AD, Philip S, Goatman KA, Williams GJ, Olson JA, Sharp PF. Automated detection of exudates for diabetic retinopathy screening. Phys Med Biol. 2007 Dec 21;52(24):7385-96. doi: 10.1088/0031-9155/52/24/012. Epub 2007 Dec 5.
• Philip S, Fleming AD, Goatman KA, Fonseca S, McNamee P, Scotland GS, Prescott GJ, Sharp PF, Olson JA. The efficacy of automated "disease/no disease" grading for diabetic retinopathy in a systematic screening programme. Br J Ophthalmol. 2007 Nov;91(11):1512-7. doi: 10.1136/bjo.2007.119453. Epub 2007 May 15.
• Abramoff MD, Niemeijer M, Suttorp-Schulten MS, Viergever MA, Russell SR, van Ginneken B. Evaluation of a system for automatic detection of diabetic retinopathy from color fundus photographs in a large population of patients with diabetes. Diabetes Care. 2008 Feb;31(2):193-8. doi: 10.2337/dc07-1312. Epub 2007 Nov 16.
• Bresnick GH, Mukamel DB, Dickinson JC, Cole DR. A screening approach to the surveillance of patients with diabetes for the presence of vision-threatening retinopathy. Ophthalmology. 2000 Jan;107(1):19-24. doi: 10.1016/s0161-6420(99)00010-x.
• Kinyoun JL, Martin DC, Fujimoto WY, Leonetti DL. Ophthalmoscopy versus fundus photographs for detecting and grading diabetic retinopathy. Invest Ophthalmol Vis Sci. 1992 May;33(6):1888-93.
• Pattichis MS, Pattichis CS, Avraam M, Bovik A, Kyriacou K. AM-FM texture segmentation in electron microscopic muscle imaging. IEEE Trans Med Imaging. 2000 Dec;19(12):1253-8. doi: 10.1109/42.897818.
• Grading diabetic retinopathy from stereoscopic color fundus photographs--an extension of the modified Airlie House classification. ETDRS report number 10. Early Treatment Diabetic Retinopathy Study Research Group. Ophthalmology. 1991 May;98(5 Suppl):786-806.
• Harding SP, Broadbent DM, Neoh C, White MC, Vora J. Sensitivity and specificity of photography and direct ophthalmoscopy in screening for sight threatening eye disease: the Liverpool Diabetic Eye Study. BMJ. 1995 Oct 28;311(7013):1131-5. doi: 10.1136/bmj.311.7013.1131.
• Kalm H, Egertsen R, Blohme G. Non-stereo fundus photography as a screening procedure for diabetic retinopathy among patients with type II diabetes. Compared with 60D enhanced slit-lamp examination. Acta Ophthalmol (Copenh). 1989 Oct;67(5):546-53. doi: 10.1111/j.1755-3768.1989.tb04106.x.

Helpful Links

• Related info.
• Related info.
• Related info.
• Related info.
• Related info.

Study record dates

Study Major Dates

• Study Start: August 1, 2012
• Primary Completion (Actual): February 24, 2015
• Study Completion (Actual): February 24, 2015

Study Registration Dates

• First Submitted: May 24, 2012
• First Submitted That Met QC Criteria: June 4, 2012
• First Posted (Estimate): June 7, 2012

Study Record Updates

• Last Update Posted (Actual): September 18, 2017
• Last Update Submitted That Met QC Criteria: September 14, 2017
• Last Verified: September 1, 2017

More Information

Terms related to this study

• Keywords: Diabetic Retinopathy; Neovascularization; Diabetes Type One; Diabetes Type Two; PreDiabetes; Retinal Screening; RiskAnalyzer; Retinal Imaging; Algorithm; Microaneurysms; Intraretinal microvascular abnormalities
• Additional Relevant MeSH Terms: Cardiovascular Diseases; Vascular Diseases; Eye Diseases; Endocrine System Diseases; Diabetic Angiopathies; Diabetes Complications; Diabetes Mellitus; Retinal Diseases; Diabetic Retinopathy
• Other Study ID Numbers: EY18280-04; 2R44EY018280-04 (U.S. NIH Grant/Contract)