2.1.

AOSLO Imaging

Videos were acquired on an AOSLO as previously described.¹ AOSLO videos can be acquired using different imaging configurations, depending on the application. We consider three different configurations of imaging parameters (Type 1, 2, and 3), as described in Table 1. Type 1 videos were acquired for the purpose of analyzing blood flow, Type 2 for analyzing photoreceptors, and Type 3 for analyzing both blood flow and photoreceptors. Type 1 videos were acquired using a green wavelength laser, which has been reported to be the optimal wavelength for obtaining good contrast in blood flow imaging.¹¹ Type 2 and 3 videos were acquired using a near-IR laser — a more desirable wavelength in terms of (i) risks due to laser damage on the retina, (ii) compatibility with the AOSLO hardware, and (iii) overall subject comfort. Type 1 and 3 videos were acquired for longer durations in order to increase the number of leukocytes that could be counted. Examples of frames from Type 1, 2, and 3 videos are shown in Fig. 2.

Fig. 2

Examples of retinal images noninvasively acquired in human subjects using an AOSLO, for a Type 1 (top row), Type 2 (middle row), and Type 3 (bottom row) video. The first column shows the averaged image of all frames for each video. The remaining three columns are three consecutive frames, showing a leukocyte (circled) traveling through a parafoveal retinal capillary. The frames in these videos have been preprocessed. The foveal center is near the top right corner for all three videos. Circular dots are photoreceptors, and dark fuzzy lines usually correspond to capillaries. However, the locations of all capillaries are not obvious. Scale bar, 100 μm.

Table 1

Imaging parameters and subject data for various types of AOSLO videos. Three representative configurations of imaging parameters used for AOSLO imaging, for three different subjects. Representative values are given for the scale factors, which vary by small amounts across different imaging sessions due to small variations in hardware alignment and eye morphology.

Since the Type 1 videos had the highest spatial contrast for the moving blood cells, we developed our methods using only Type 1 videos, and used Type 2 and 3 videos as well as synthetic videos generated by a virtual AOSLO for verification.

2.2.

Preprocessing

Raw videos were preprocessed to correct for distortions due to raster scanning and eye motion, without considering object motion. Preprocessing involves desinusoiding, stabilization, cropping, and frame deletion.

2.2.3.

Frame deletion

In the processed videos, there were three types of improper frames that were identified for deletion. First, insufficient overlap between the image and the reference frame resulted in poor stabilization. This occurred when the eye wandered too far away from its fixation target. Second, blinks resulted in the image intensity dropping to zero throughout the blink. Third, large saccades, involuntary fast eye movements, caused intraframe shearing and distortion on single frames and prevented proper image stabilization (1). To generate high quality images of photoreceptors and vessels, these “improper” frames were deleted. However, we did not delete any frames for the speed analysis since deletion of frames would increase the apparent speed of a moving object.

Video 1

Type 1 AOSLO video, slowed to 5 fps (real time frame rate is 30 fps). An example of an improper frame resulting in poor registration is at 0:03 s into the video. (MPEG, 3.73 MB).1

10.1117/1.3548880.1

2.3.

Visualization of Moving Objects and Vessels

Since spatial contrast is low, motion contrast enhancement is used to visualize moving objects and vessels (Fig. 3). Methods for motion contrast enhancement have been previously described. ^{6, 12, 13, 14} We implement a method that works well with AOSLO videos,⁶ using a multiframe division video and a standard deviation image. The multiframe division videos were used to visualize moving objects and the standard deviation image was used to visualize vessels. Median and Gaussian filtering were applied before and after calculation of the multiframe division video, respectively.

Fig. 3

Visualization of vessels and moving objects by applying motion contrast enhancement to the Type 1 (top row), Type 2 (middle row), and Type 3 (bottom row) AOSLO videos shown in Fig. 2. The first column shows the standard deviation image, which enhances the contrast of vessels. The next three columns show three consecutive frames of the multi-frame division video, which enhances the contrast of moving objects. Scale bars, 100 μm.

A preprocessed video has moving blood cells in front of a stationary background tissue, consisting of photoreceptors and vessels. Given two frames, I _j(x,y) and I _j+1(x,y), the division image [TeX:] $D_j \left({x,y} \right) = \frac{{I_j \left({x,y} \right)}}{{I_{j + 1} \left({x,y} \right)}}$ $D_j\left(x,y\right)=\frac{I_j\left(x,y\right)}{I_{j+1}\left(x,y\right)}$ emphasizes the objects in motion as long as the intensity of background tissue remains relatively constant. Here, I _j(x,y) represents the intensities of frame j at position (x,y). Division images are used instead of difference images to enable arithmetic averaging of multiple frames, which improves the signal to noise ratio, as opposed to using the arithmetic average of two consecutive difference images, which yields no improvement in signal to noise.¹⁵ We defined a multiframe division video as [TeX:] $M_j \left({x,y} \right) = \frac{{D_j \left({x,y} \right) + D_{j + 1} \left({x,y} \right)}}{2}$ $M_j\left(x,y\right)=\frac{D_j\left(x,y\right)+D_{j+1}\left(x,y\right)}{2}$ .

To visualize the perfused vessels, an image was calculated from the multiframe division video, using the geometric standard deviation. For a video with n frames, the geometric standard deviation image, S(x,y), is defined as

2.4.

Object Tracking

We used cell paths on ST plots for object tracking.¹⁵ ST plots are a method to visualize hemodynamics and offer two major advantages. First, the ST plot representation is more compact, which assists in pattern identification. Variables such as density, frequency, and variations in speed both spatially and temporally can easily be observed on ST plots, but not from direct examination of the original video. Second, the dimensional complexity of the problem is reduced from a 4-dimensional (3D+1T) problem to a 3-dimensional (2D+T) problem, which minimizes the computation cost. These plots have been used for many systems, ^{12, 16, 17, 18} including AOSLO systems. ^{15, 19}

ST plots were generated by converting an X-Y-T coordinate system into an s-T coordinate system (Fig. 4). Consider an arbitrary vessel in a sequence of frames, with an object that moves along the trajectory of the vessel, given by f(x,y). By plotting intensity values along the vessel, and discarding all other pixel values, a two-dimensional plot can be generated that shows the movement of individual objects traveling through a one-dimensional line, given by f(s), where f(x,y) ↔ f(s) is naturally defined, with the first coordinate of f(x,y) mapping to the first element of f(s), the second coordinate to the second element, and likewise for the remaining elements. Since we are exactly specifying the mapping from each pixel in X-Y-T space to s-T space, the mapping is invertible.

Fig. 4

Conversion of X-Y-t coordinate system (left) into s-t coordinate system (right). Three consecutive frames, (k−1, k, k+1), are shown with a dark circle representing a single leukocyte traveling along a vessel centerline.

For single-file flow through capillaries seen by AOSLO videos, there is no loss in speed information when switching from the X-Y-T representation to the s-T representation.¹⁵

Motion contrast enhancement improves the ST plots by increasing the accuracy of vessel centerline extraction and by increasing the contrast of cell paths.¹⁵ Using motion contrast enhanced ST plots, we manually extracted cell traces for the tracking and speed quantification. To identify traces, the user was presented with a graphical user interface showing a portion of a ST plot. The user identified traces by selecting points along that trace. For consistency, points were selected at the border between the dark and bright regions of the trace, on the leading edge (Fig. 5). After points were selected, interpolation was performed using piecewise splines constrained to the pixel resolution.

Fig. 5

Generation of motion contrast enhanced ST plots from AOSLO videos using the same videos shown in Fig. 2. The first column shows the vessel centerline that was selected for analysis. The second column shows ST plot analysis. On the ST plots, diagonal streaks represent moving objects, while vertical streaks represent blinks or saccades, where the video intensity drops for 1–3 frames. Extracted traces are directly shown beneath the corresponding ST plot. Type 1 video (top), all frames (1 to 1200), and frames 201 to 400 magnified from entire strip. Type 2 video (middle), all frames (1 to 126). Type 3 video (bottom), all frames (1 to 2387), and frames 1987 to 2387 magnified from entire strip. Vessel image scale bars, 100 μm; ST plot horizontal scale bars, 0.5 s, vertical scale bars, 0.25 mm.

For tracking, the coordinates of each extracted trace were used to register the location of the blood cells in the video. ST coordinates (s, t) were converted back to video coordinates (x, y, t) using the invertible mapping defined during generation of ST plots. Video coordinates were compiled into a list and then used to mark object locations to visualize the tracking results.

2.5.

Quantification of Object Speeds

In the absence of eye motion, the speed of an object in a raster scanning system can be explicitly computed using line information from pairs of frames.²⁰ The correction is based on computing the actual time at which a given line was acquired, as opposed to assuming that the entire frame was acquired at the same time. The true time, t, can be computed as

In a nonraster scanning system without eye motion, the speed of an object can be computed by simply computing the slope of the extracted trace from an ST plot. However, in our system, the slope of the trace gives speeds in time units of frames. We present a slope modification procedure to correct speeds, based on computing the acquisition time coordinates in the extracted traces. In order to perform the correction, line numbers on preprocessed videos need to be transformed back to line numbers on the raw videos (not preprocessed). This is important because the correction for intraframe eye motion results in local stretching or compression of pixels, thereby altering line numbers.

The AOSLO uses a fast horizontal scan and a slower vertical scan, from left to right and top to bottom directions, respectively. The main component of the error is due to the slower vertical scan; the error due to the horizontal scan is small and does not need to be corrected (Secs. 2.6, 3.1). As an example, in the absence of eye motion, a downward moving object will have a larger observed displacement compared to the actual displacement, since the scan is chasing a moving target. More generally, any object that is moving in a nonhorizontal trajectory has a vertical component of velocity that needs to be corrected. If there is eye motion, then the actual displacement is also dependent on the amount that the eye has moved.

Consider coordinates from the extracted traces, given as (frame number, s). The acquisition time for each line (in units of partial frames) can be computed as

L is the line number at which the data was taken on the raw video, and can be recovered in the following manner:

Compute L = S ⁻¹(L_crop + dL_crop). The extracted traces are then plotted as (acquisition time, s).

For each corrected trace, a linear regression was applied and the slope of the line, with units of pixels/frame, was used to compute the speed of the leukocyte (in units of mm/s) through the selected vessel segment in the following manner:

2.6.

Expected Error due to Raster Scanning (RS)

The raster scan error is significant for AOSLO videos. In this section, we develop a theoretical model to quantify the magnitude of the raster scan error. To understand the nature of the expected raster scan error, consider the case of a vertically oriented vessel with a downward-moving object that starts at the top of the image (Fig. 1). Assuming that there is no eye motion, we derive the expected raster scan error in the vertical and horizontal cases for comparison to actual measured error rates and show that it is significant in the vertical direction, but not the horizontal direction.

We introduce the dimensionless number,

If RS > 1, then the system is unable to image the object and the error becomes infinite. When RS = 0.5, the error is exactly 100%. When v _o ≪ v _s, RS → 0, and the raster scan error is negligible. By convention, leukocyte speeds on the retina are reported in mm/s. For an object speed given in mm/s, [TeX:] $\overline {{v}_o }$ $\overline{v_o}$ , with scan speed, v _s, given in pixels/frame, with scale factor as defined in Table 1, RS can be calculated as

Fig. 6

Expected raster scan error assuming no eye motion, for a vessel that is oriented in the same direction as the scanner. The range of errors for an object moving between 1 and 3 mm/s is shown for a Type 1 (T1), Type 2 (T2), and Type 3 (T3) AOSLO video. The portion of the plot corresponding to each video type is marked.

Thus, the raster scan error increases when one or more of the following occur: v _s decreases, [TeX:] $\overline {{v}_o }$ $\overline{v_o}$ increases, or the field of view decreases (the field of view inversely varies with the scale factor in the scan direction).

A similar analysis can be used to estimate the expected error in the horizontal scan direction. Since the scan speed is defined as the number of rows required to reach the edge of the frame, for the horizontal direction, RS _horizontal = RS _vertical/512. The percent error in the horizontal direction = 100%^* [TeX:] $\frac{{RS_{\rm vertical} }}{{512 - RS_{\rm vertical} }}$ $\frac{R{S}_{\mathrm{vertical}}}{512-R{S}_{\mathrm{vertical}}}$ . Since RS _vertical < 1 for objects of interest, the percent error in the horizontal direction will always be less than 0.2%. When RS _vertical = 0.5, the percent error in the horizontal direction drops to 0.098%. Therefore, we do not need to apply the raster scan correction to the horizontal component of calculated speeds.

2.7.

Virtual AOSLO

The AOSLO is a custom-built, unique instrument with ∼5× the resolution of a commercial scanning laser ophthalmoscope. Typically, ground truth for new systems is generated using manual analysis performed by subject experts. However, due to the low contrast of the moving objects, it is difficult and unreliable to analyze videos by naked eye. Therefore, we used a virtual AOSLO to simulate realistic videos to create a synthetic dataset for use as ground truth in order to validate our methods.

2.7.1.

Parameters

A virtual AOSLO has previously been used to characterize scanning distortions due to raster scanning and eye motion for static images.²² We modified this virtual AOSLO to simulate the acquisition of a video in the presence of an object that is moving at the same time as the scanner. For the virtual AOSLO, we selected scanning parameters for a Type 1 video, which is the data set on which the proposed methods were developed. Since we were able to exactly specify the speed and position of the moving object, we considered the simulated videos to be ground truth. Due to the complexity of the AOSLO, the following assumptions were used for the virtual AOSLO:

• 1 The imaging laser is a perfect, dimensionless dot that samples with true fidelity one pixel of the input image at one time.

• 2 The retina is rigid and planar across the field of view.

• 3 Eye motion is strictly translational with no torsional component.

• 4 Imaging parameters are constant.

The first assumption bypasses sampling and resolution issues introduced by the optics of a human eye. This means that the quality and appearance of the simulated video primarily depends on the input image. Second, for the field of view of the simulation (1.5 deg in each direction), it is reasonable to assume that this region is both rigid and planar. Third, as previously described, the primary components of eye motion are translational.⁷ While we have observed torsional motions, they are typically small (unpublished experimental observations). The final assumption is that imaging parameters are constant — in actual practice, due to additional complexities such as calibration and temperature-dependent drift of electronic components, different imaging sessions have minor variations in imaging parameters (these variations are addressed using calibration steps prior to each imaging session).

An overview of the virtual AOSLO is shown in Fig. 7, and a summary of the parameters selected for the simulation is shown in Table 2. The input image was generated using individual frames from overlapping AOSLO videos near the fovea, which were scaled to the appropriate size (circle diameter = 3.75 deg). The spatial resolution of the input image was selected to be twice that of the output video and sampled by the virtual AOSLO using nearest neighbor interpolation. Thus, the simulated videos are similar to actual AOSLO videos, which allow us to apply the same correction steps that we would have applied to actual AOSLO videos.

Fig. 7

Overview of virtual AOSLO used to simulate videos of a moving object in the presence of raster scanning and eye motion. An arbitrary vessel and flow direction was inserted into a retinal input image and an object speed was specified. The rigid input image, vessel, and object were translated according to specified X and Y Eye Motion (EM) inputs. The frames in the output video were generated using the scan parameters and configuration described in Sec. 2.7.1. Scale bars, 100 μm.

Table 2

Summary of parameters for the virtual AOSLO.

Parameter	Value
Horizontal raster frequency	15.36 kHz
Vertical raster frequency	30 Hz
Video frame size	512×512 pixels
Video acquisition rate	30 fps
Sampling band	Central 80% of forward sweep
Retinal scale factor	0.296 mm/deg
X and Y scale factors	341.33 pixels/deg
Vessel diameter	5 μm
Leukocyte length	15 μm in the direction of travel
Leukocyte speeds	1.00 to 3.00 mm/s

Simulated videos were generated pixel by pixel. The basic steps were to 1. calculate the pixel timing given the raster parameters, 2. convert timings to spatial coordinates on the input image, and 3. sample the input image at the specified spatial coordinates. The time that each pixel is sampled can be directly computed using the raster scan parameters. To convert times to spatial coordinates, two calculations were done. First, the spatial coordinates were calculated assuming a static image. Second, the spatial coordinates were horizontally and vertically translated as specified by the X and Y components of eye motion, corresponding to the timing at each pixel location. Finally, sampling was performed after insertion of a moving object into the input image. For the moving object, we specified the trajectory and speed and used a high contrast, elongated object oriented along the direction of travel. The time-dependent input image was then sampled at the corresponding spatial coordinate for each pixel to generate a simulated AOSLO video.

3.1.

Evaluation of Accuracy and Validity Using a Virtual AOSLO

We applied the proposed methods for tracking and speed quantification (Fig. 8). To measure speeds in the simulated videos, we repeated the analysis five times and took the average of computed speeds, in order to reduce the errors due to operator bias and differences in data precision. The data precision varied since there were three times as many data points that could be extracted to measure speeds at 1 mm/s versus 3 mm/s. At 3 mm/s, due to large pixels/frame displacements, the traces on the ST plots were disconnected.

Fig. 8

Tracking and speed quantification on V1_EM. (A) First frame in which the object appears from a preprocessed video. (B) Corresponding frame in the multiframe division video. (C) Standard deviation image calculated from 15 frames. (D) Extracted vessel for offline generation of the ST plot. (E) ST plot for the selected vessel. (F) Extracted trace from the ST plot (dotted line on left) and the corrected trace taking into consideration both raster scanning and eye motion (solid line on right). Notice that the speed was overestimated prior to correction, as expected. (G) Three consecutive frames of the video showing tracking results.

To validate that objects were being correctly tracked, we generated a tracked video and used frame-by-frame examination. As expected, for all videos, the extracted traces corresponded to the moving objects. However, the labeled lines would sometimes lead or lag the moving objects by small amounts. Since the amount of lag/lead was preserved for each moving object, the slope of the traces was accurate. The error was due to the estimation of frame number from the coordinates of the extracted traces, due to the low temporal resolution relative to the speed of the leukocytes. Taking this into consideration, there were no false positives and no false negatives.

We compared the corrected speeds to the actual speeds (Table 4). We define the residual error as the percent difference between corrected and actual speeds and found that the residual error was on average 2% for moving objects traveling between 1 and 3 mm/s. The sources of error are most likely due to vessel and trace extraction, which are dependent on user interaction. For experimental data, these sources of error are likely to increase due to 1. lack of prior information about vessel trajectories and 2. variations in trace slopes. For the synthetic data sets, extraction is more accurate due to prior knowledge about the shape of the vessel (since it was specified), and due to the fact that object speeds are uniform (so that there is no variation in trace slopes).

Table 4

Evaluation of speed quantification. Speeds are reported before (no RS) and after (RS) the proposed correction. Actual speeds are the object speeds corresponding to each video, as listed in Table 3.

H3 confirms that the error in measured speed is negligible in the horizontal direction, since the calculated error was −0.09%. This is also in agreement with the theoretical model, which specifies an upper bound of 0.2% for the error. Therefore, it is a reasonable assumption to neglect the error due to horizontal scanning.

V1 and V3 confirm the theoretical errors due to the vertical component of raster scanning. We found errors of 8.31 and 28.63%, which are in agreement with the theoretical errors of 8.1 and 29.1%. Therefore, in the absence of eye motion, the computed errors are in agreement with the expected errors.

Eye motion can either increase or decrease the magnitude of the error. If eye motion is random and isotropic, then over time the average speed should not be affected by eye motion. However, if the eye favors motion along a preferred direction, then the computed speed is affected — the computed speed is maximally increased when the object, raster scan, and eye motion are in the same direction (i.e., all vertical and downward). Initially, the vertical component of the eye motion trace input is in the same direction of the scan — as expected, the error for V1_EM is slightly larger than V1.

In practice, vessels are rarely horizontal or vertical, particularly when considering capillaries. First, the magnitude of the error in calculated speed depends on the trajectory of the vessel at the object location, since only the vertical component of speed is corrected. Therefore, deviations from a vertically-oriented vessel should result in diminishing error magnitudes. Second, the start and end points of the vessel ultimately determine whether speeds are over- or underestimated. The vessel in A2 and A2_EM has both up- and downward components, but since the endpoint is lower than the starting point, this means that we should expect speeds to be underestimated when comparing A2 to V2. Since the eye motion results in a slight overestimation (comparing V1_EM to V1 and using the same eye motion input), this explains why the error for A2_EM is less than the error for A2.

3.2.

Evaluation on Experimental AOSLO Videos

We performed the proposed methods on 40 vessels from ten AOSLO videos; first we report results across all videos and then we show detailed results for one representative vessel for each video Type.

Ten vessels were analyzed from 1 Type 1 video, ten vessels from 3 Type 2 videos, and 20 vessels from six Type 3 videos. The average absolute error in measured speed was 2.59% for the Type 1 video, 3.39% for the Type 2 video, and 2.04% for the Type 3 video, where absolute error was defined as the absolute value of the percent difference between corrected and noncorrected speeds for one trace, and the average absolute error was defined as the average absolute error across all extracted traces for each video Type. For comparison, we estimated the error using the RS parameter as defined in Sec. 2.6, taking [TeX:] $\overline {{v}_o }$ $\overline{v_o}$ to be the average object speed. In the absence of eye motion, for a vertically-oriented vessel, the theoretical error was 12.56% for the Type 1 video, 12.84% for the Type 2 video, and 5.27% for the Type 3 video. This suggests that either vessel orientations are horizontally biased or that eye motion is not uniformly distributed across all orientations.

We selected three representative vessels from Type 1, Type 2, and Type 3 videos to further characterize the error in measured speeds. For each vessel, traces were extracted from ST plots and used for tracking on the original videos and speed quantification (Fig. 9). Close examination of Fig. 9 shows that the orientation of the vessel has an effect on the slope modification as long as the effect due to eye motion is small (i.e., one can see whether the slope was over- or underestimated corresponding to a downward- and upward-oriented vessel). We will discuss this effect in more detail considering the actual errors in average speeds that were calculated (Table 5). In experimental videos, there are complexities such as arbitrary vessel shapes and orientations, eye motion, and variations in cell speeds both temporally and spatially. There is also noise due to variations in the intensity of the background photoreceptor tissue, likely due to dynamic scattering changes over time²³ and also coherent artifacts. ^{24, 25} These variations generate noise in the multiframe division videos and affect the appearance of the ST plots. Therefore, the actual error in calculated speeds due to raster scanning and eye motion will be different. We compared our corrected speeds to uncorrected speeds, where uncorrected speeds were simply taken as the slope of the manually extracted trace, which assumes that the entire frame was acquired at the same moment in time.

Fig. 9

Effect of raster scan showing extracted traces (dotted lines on left) and raster scan corrected traces (solid lines on right), for Type 1 (T1, frames 800–830), Type 2 (T2, frames 48–60), and Type 3 (T3, frames 2278–2302) videos. The shifts are nonuniform due to the confounding effects of eye motion. Vertical scale bars, 0.1 mm; horizontal scale bars, 0.1 s.

Table 5

Summary of cell speeds in selected vessel segments with and without the raster scan correction.

The actual error in the average speeds is 2.51% for the Type 1 video, 9.25% for the Type 2 video, and 2.83% for the Type 3 video. As previously described, the magnitude and sign of the error is largely determined by the trajectory of the vessel. For all three types, the vessels deviate from a purely vertical vessel, and so the magnitude of the error is diminished compared to the model (Sec. 2.6). In addition, because the end point of each vessel is lower than the start point, we expect an overestimation of speed for all three Types. The Type 2 video has the largest error, as predicted by the theoretical model (Fig. 6). Notice that there is a nonlinear shift that results due to the raster scan correction. Close examination of the Type 1 traces in Fig. 9 show that the slopes at the bottom of individual traces were decreased after application of the raster scan correction, while slopes at the top were increased. This suggests that the entrance side of the path segment underestimated speeds, while the exit side overestimated speeds, corresponding to a net upward and net downward vertical orientations, respectively. As can be seen from Fig. 5, this was exactly the case. As a final comparison, for the Type 1 video, the error was 2.51%, compared to −1.07% for the same vessel in A2_EM. The reason for this is due to a small difference in the starting and ending points of the vessel. Although we used the same trajectory, the end point of the vessel slightly terminates higher than the starting point for A2_EM, which explains the difference in the sign of the error.

To verify that each extracted trace corresponded to an object on the input video, extracted traces were registered (Sec. 2.4; illustrated in Fig. 10). We individually verified each extracted trace by examining the tracked video frame-by-frame. Overall, the labeled lines tracked the leukocytes well. There were no false positives; it was not possible to calculate a false negative rate.

Fig. 10

Moving objects labeled using white lines in the cropped video using extracted traces from ST plots, for a Type 1 (top), Type 2 (middle), and Type 3 (bottom) video. For visualization purposes, we thickened the line by 2 pixels. Scale bars, 100 μm.