Article Text
PDF
Abstract
Aim To demonstrate the potential clinically meaningful benefits of a thermal camera integrated with the Argus II, an artificial vision therapy system, for assisting Argus II users in localising and discriminating heat-emitting objects.
Methods Seven blind patients implanted with Argus II retinal prosthesis participated in the study. Two tasks were investigated: (1) localising up to three heat-emitting objects by indicating the location of the objects and (2) discriminating a specific heated object out of three presented on a table. Heat-emitting objects placed on a table included a toaster, a flat iron, an electric kettle, a heating pad and a mug of hot water. Subjects completed the two tasks using the unmodified Argus II system with a visible-light camera and the thermal camera-integrated Argus II.
Results Subjects more accurately localised heated objects displayed on a table (p=0.011) and discriminated a specific type of object (p=0.005) presented with the thermal camera integrated with the Argus II versus the unmodified Argus II camera.
Conclusions The thermal camera integrated with the artificial vision therapy system helps users to locate and differentiate heat-emitting objects more precisely than a visible light sensor. The integration of the thermal camera with the Argus II may have significant benefits in patients’ daily life.
- vision
- retina
- prosthesis
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Blindness affects 36 million people worldwide, and to date, the only US Food and Drug Administration (FDA)-approved artificial vision device is the Argus II retinal prosthesis system (Second Sight Medical Products; Sylmar, CA, USA).1 The Argus II prosthesis is approved for blind patients with end-stage retinitis pigmentosa (RP), which is a group of conditions that affects around 1 in 4000 people to varying levels of severity. The current US and European indication are for adults 25 years or older with bare light or no light perception vision in both eyes and a prior history of useful vision.2 The Argus II allows patients to detect motion3 and perform spatial-motor tasks,4 but the patient’s performance with the Argus has a large variation, whereas some can recognise letters, others can only detect and localise high contrast targets.5 There are approximately 350 Argus II users in the world.6
Despite the novelty of the Argus II, it is extremely limited by the resolution of imagery that it can provide to users. The Argus II provides flashes of light to the wearer by translating images captured through an external camera to electrical stimulation of a 60-electrode epiretinal array, and it relies on various brightness levels within a captured scene to determine the stimulation amplitude of the different electrodes. Thus, in scenes that lack sufficient difference in brightness, such as a beige object against a white background, the prosthesis is unable to provide meaningful imagery to the patient. Additionally, in scenes with multiple small objects of varying brightness, it may be difficult for wearers to interpret the low-resolution image that the prosthesis provides. To simplify the image that the user sees, previous studies have identified potential advantages of a thermal-integrated Argus II system in allowing wearers to better locate heat-emitting objects (abstracts presented at ARVO annual meetings and they are listed in https://arvojournals.org/. Sun et al. Investigative Ophthalmology & Visual Science July 2019, Vol 60, 4978. Bajaj et al, Investigative Ophthalmology & Visual Science July 2019, Vol 60, 4017. Dagnelie et al, Investigative Ophthalmology & Visual Science September 2016, Vol 57, 5167).
In this study, we further explored the role of the thermal-integrated Argus II system in aiding prosthesis wearers interact with common heat-emitting household objects, which could be an important household safety feature of the prosthesis. We compared the two cameras, the current Argus II system that depends on the light in the visible spectrum (in this paper we refer as the ‘Argus II camera’) versus the thermal sensor camera integrated to the Argus II system (in this paper we refer as ‘thermal camera’. Given that the thermal camera can detect objects above a certain temperature, we hypothesised that a thermal camera integrated with an artificial vision therapy system, the Argus II, is better able to help users locate and identify the specific type of heat-emitting household object in comparison with the visible-light camera of the existing Argus II system.
In this paper, we chose the term of ‘artificial vision therapy’ as a more general term that implies that this research is not only applicable to the Argus II, but for other emergent visual prosthesis systems (eg, Orion cortical visual prosthesis by Second Sight). It also implies that the type of vision restored by a visual prosthesis is different from the ‘normal’ vision, requiring a substantial amount of training and rehabilitation to use the technology effectively.
Methods
Participants
A total of seven blind subjects (five men and two women; age 66.9±9.3 years) with a history of advanced RP were recruited for this study. All subjects underwent Argus II retinal implantation at the University of Minnesota and had been using the Argus II retinal prosthesis system for at least 6 months at the time of the study. Informed consent was obtained from all subjects. The study protocol was approved by the University of Minnesota Institutional Review Board and adhered to the tenets of the Declaration of Helsinki.
Thermal camera integration and Argus II setup
The integration of the thermal camera with the Argus II system was described in an earlier study.7 In summary, a forward-looking infrared radiometer (FLIR) Lepton camera (FLIR Systems Inc; Wilsonville, Oregon, USA) was selected as the thermal camera. The FLIR camera, mounted on a pair of eyeglasses, was wired to a thermal processor based on Zynq 7010 system on module (Xlinx, San Jose, California, USA). The output of the thermal processor was routed to the Argus II system. Thus, the FLIR camera was used in lieu of the standard Argus II system camera to create the thermal camera-integrated Argus II system.
In general, the thermal system maps temperature to greyscale. Hence, the ability to discriminate between targets at a different temperature such as heat-emitting objects or the human body versus the background depends on the ability of the patient to discriminate between greyscales. The conversion between temperature and greyscale is set to enhance the contrast in temperature around the heat emitting object or human body. For instance, a temperature below 28°C is mapped to black and temperature above 35°C is mapped to white with linear scaling in between. This mapping enhances the detectability of humans. In a similar manner, we adjusted the mapping for the objects evaluated in this study.
All subjects’ Argus II settings were previously fitted and optimised per standard Second Sight protocol and were not altered for this experiment.
Study tasks
Two tasks involving subject interaction with heat-emitting household objects were used to compare the ability of the thermal camera versus the visible-light Argus II camera in providing end-users functional vision in a real-world setting. The goal of the localisation task was to assess the subject’s ability to detect the presence of objects, and the discrimination task focused on subject proficiency in distinguishing between different types of objects commonly used in daily life.
Five heat-emitting common household objects were selected for this study, which included a toaster, a flat iron, a mug of hot water, a tea kettle and a heating pad. The temperatures of these items were measured with a digital multimeter (Klein Tools MM400 Digital Auto-Ranging Multimeter, Lincolnshire, Illinios, USA) and ranged from 105°F to 258 °F. To help subjects become more familiar with objects used in this study, they used their artificial vision to examine and verbally describe each of the objects, and then felt the shape of each object using thermal protective gloves prior to performing the study tasks.
Subjects used their artificial vision therapy system to complete the following study tasks:
Heat-Emitting object evaluation: localisation
A table covered with white table cloth was used (figure 1). Up to three of the five objects were placed on the table in three possible locations, left, centre and right, during each trial. Subjects completed 12 trials with each the thermal camera and the Argus II camera, including 4 one-object trials, 4 two-object trials and 4 three-object trials. The trial sequence and the position(s) of object(s) placement during each trial were predetermined by a random sequence generator.
The set-up table for the experiments. A subject using his Argus II prosthesis was seated in front of a table covered with a white cloth to look for up to three heat-emitting objects. For task one, the subject was asked the number and the position(s) of the object(s). For task two, the subject was asked to discriminate a specific object out of the three presented.
The subjects sat in front of the table and counted the number of objects they saw on the table using their artificial vision. Additionally, they identified the location (left, centre and right) of each object on the table. The time required to complete each trial was recorded. A maximum of 3 points was awarded for each trial, with one point given for correctly determining the presence or absence of an object at each of the three locations.
Heat-emitting object evaluation: discrimination
In each trial, three of the five objects were selected by a random sequence generator and placed on the left, centre and right of the table in front of the subjects. The subjects were asked to determine the location of a specific item (randomly predetermined for each trial) on the table. For example, a flat iron, a mug of hot water and a heating pad could be placed on the table, and the subject would be asked to locate the flat iron. One point was awarded for the correct answer in each trial. Subjects completed 10 trials with each of the thermal camera and the Argus II camera.
Data analysis
Subject scores (detailed in Study Tasks section) from each activity as well as time to completion of each task were analysed. Paired t-tests were used to assess differences in results between the thermal camera and the Argus II camera. A p value <0.05 was considered statistically significant.
Results
In the localisation task, the subjects scored an average of 1.6 points per task out of 3 possible points (53% accuracy) with the Argus II camera and 1.9 points (63% accuracy) with the thermal camera (p=0.011). Individually, six out of seven subjects performed better with the thermal camera, while one subject achieved the same score with the two devices (figure 2A).
Results for (A) object localisation and (B) object discrimination. Left: the average trial score is shown for each subject. For object localisation, the maximum score for each trial is three points and for discrimination, the maximum score per each trial is one point. Right: the average score across all subjects. Error bars indicate ±1 SEM. P-values <0.05 were considered statistically significant. *p<0.05; **p<0.01.
In the discrimination task, subjects overall performed better with the thermal camera. They scored an average of 0.2 points per trial (20% accuracy) with the Argus II and 0.5 points (50% accuracy) with the thermal camera (p=0.005). Like the localisation task, six of seven subjects received higher scores with the thermal camera system, whereas one subject achieved the same score with two devices (figure 2B).
On average, there was no significant difference in the amount of time that subjects took to complete either task with the Argus II camera or the thermal camera. The average time for completing the localisation task with the Argus II was 57±31 s versus 49±32 s with the thermal camera (p>0.05). Additionally, subjects needed an average of 43±26 s with the Argus II versus 48±30 s with the thermal camera to complete the discrimination task (p>0.05).
Discussion
The two tasks developed for this study, localisation and discrimination, mimic ways that patients could potentially use their artificial vision therapy system to interact with different heat-emitting household objects.
Overall, the subjects performed statistically better with the thermal camera in comparison with the Argus II camera. In this experiment, the only warm objects in the visual field were those on the table. These objects generated a high-contrast image with respect to their surroundings when using the thermal camera. For the Argus II camera, the observer had to discriminate between the objects on the table and everything else in the visible light spectrum. This included objects in the background, not only on the table, for example. Thus, the thermal camera provided a ‘de-cluttered effect’ compared with the Argus II camera that could provide an overwhelming sensory input of the objects and background. The low accuracy response for the Argus II camera could also be due to the lack of contrast between the objects placed on the table and the background. The objects were chosen to simulate real-life conditions.
With respect to the visual experience provided by the thermal camera, since the visual input captured by either camera travels in the same manner to the electrode array, the subject perceives phosphenes or patterns elicited by the electrical stimulation. As an analogy, the input receiving with the thermal camera could be like the Argus II camera when the contrast is high, such as a white plate on a black tabletop. In addition, because it is a thermal sensor camera and not a visible-light camera, it can work in light conditions and dark conditions.
Although further investigation is necessary to improve both the Argus II and the thermal camera, it is important to recognise the better performance of the thermal camera in aiding wearers to localise and discriminate heat-emitting household objects. The improvement in performance was not merely a trade-off between accuracy and time, as the time to complete the tasks did not differ between the two camera conditions. In addition, it is important to consider that none of the subjects has had significant prior interaction or training with the thermal camera aside from participation in another study involving the thermal camera.7
At the end of our study, a survey was provided asking our subjects if they think it would be useful to integrate the thermal camera to the Argus II system. Six out of seven subjects thought it would indeed be very useful and would increase the daily use of the Argus II. Two of the six subjects found the thermal camera very useful and stated that they would prefer the thermal camera over the Argus II camera and would use it more than the Argus II camera. Only one subject mentioned that he did not feel that the thermal camera was superior to the Argus II camera in localising and discriminating heating-emitting objects, despite that his performance was better using the thermal camera (subject 1 of figure 2). This could be secondary to the lack of experience in using the thermal camera. In terms of the applicability of the thermal camera, six out of seven subjects responded that they would use the thermal camera to discriminate and localise heat-emitting household objects like stoves, fireplaces, hot cups, and so on and to help with navigation, localisation and discrimination of people and pets.
The limitations of this study include the low numbers of subjects involved in the study and the lack of long-term use of the thermal camera compared with the Argus II camera.
Currently, Second Sight Medical Products does not plan to do a commercial launch of another Argus II release after Argus 2 s. The utility of our study would be to provide an insight into the usability and advantages (performance) of this technology on a profoundly blind population targeted by visual prostheses. Based on the results of these (and other) studies, the thermal technology could potentially be integrated into future retinal or cortical systems such as the Orion (an investigational cortical visual prosthesis system by Second Sight, with FDA approval for human clinical feasibility study).
In summary, our experiments demonstrated that the thermal camera is superior to the Argus II in locating and discriminating heated objects. The thermal camera could be a valuable addition to any artificial vision therapy system and its performance could improve after more training and potentially bring greater benefits. We believe that the result of our study can be applied not only for Argus II patients but for any other emergent visual prosthesis technology.
In this experiment, we used a prototype version of the thermal camera-integrated Argus II that did not allow a transition between the thermal camera and the visible camera. That is, only one camera can be used at any time, but not both at the same time. Further studies could be done to explore the additional benefits from a truly integrated thermal-visual camera system where subjects could either easily transition between the two cameras, or the information from the two cameras is fused together. Future studies could also explore the role of the thermal camera in subject interaction with other humans, such as locating people in a room or walking through a crowd of people. Moreover, the thermal sensor can be used with an eye tracker to enable users to scan using combined eye-head movements.8
Acknowledgments
The authors would like to thank Paul Gibson from Advanced Medical Electronics for his help on various technical issues regarding the thermal camera.
Footnotes
Twitter @YingchenHePhD
Presented at Part of the work was presented at ARVO 2019 in Vancouver, British Columbia.
Contributors SRM, SS and YH: guarantor of integrity of the entire study, developed study concept, developed study design, defined intellectual content, conducted literature research, data acquisition, analysis and interpretation of data, manuscript preparation, manuscript editing, manuscript review and final approval of the version to be published. AR, AC, and JDD: guarantor of integrity of the entire study, developed the study concept and design, manuscript preparation, manuscript editing, manuscript review and final approval of the version to be published. All authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. SRM and SS: contributed equally to this work.
Funding This study was funded by the University of Minnesota Center for Applied Translational Sensory Science, Minnesota Lions Vision Foundation and VitreoRetinal Surgery Foundation.
Competing interests AR and JDD are employed by Second Sight Medical Products. AC is a consultant to Second Sight Medical Products. SRM was surgical proctor consultant to Second Sight Medical Products.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. Deidentified participant data was used in this study.
Linked Articles
- At a glance