Perhaps I can best illustrate by imagining what I should most like to see if I were given the use of my eyes, say, for just three days. And while I am imagining, suppose you, too, set your mind to work on the problem of how you would use your own eyes if you had only three more days to see. If with the oncoming darkness of the third night you knew that the sun would never rise for you again, how would you spend those three precious intervening days? What would you most want to let your gaze rest upon?
—— Helen Keller

In our vibrant world, we often lament Helen Keller’s blindness and are deeply moved by her longing for the beauty she could never see. However, the reality is that over 2.2 billion people suffer from visual impairment or blindness worldwide. Advances in visual cortex restoration technologies, particularly visual cortical prosthetics, offer a glimmer of hope for these individuals. Yet, even if partial vision is restored, can we truly understand what their world looks like?

Currently, three clinical trials of visual cortical prosthetics are underway. One uses surface electrodes (the Orion device by Second Sight Medical Products 1,2), while the other two employ deep electrodes 3,4. However, predicting the extent of restored vision remains a challenge. Furthermore, the field of neural implants heavily depends on intuition and experience, which introduces the risk of perceptual biases due to subjective interpretations.

A recent article in Scientific Reports 5 by Ione Fine and Geoffrey M. Boynton from the University of Washington introduces a “virtual patient” model grounded in the architecture of the primary visual cortex (V1). This model accurately predicts perceptual experiences reported in various studies on human cortical stimulation. Simulations reveal that increasing the number of smaller electrodes does not always improve visual restoration. In the near term, the quality of perception provided by cortical prosthetics seems more constrained by the visual cortex’s neurophysiological structure than by technological limitations.
 

1. Advances and Challenges in Vision Restoration Technologies

Vision restoration technologies are advancing rapidly worldwide, with at least eight teams developing retinal electronic implants. Among these, two devices have been approved for patient use 6-12, while others are currently in clinical trials 13. Another promising approach is optogenetics, which has shown limited but encouraging results in early clinical trials. Gene therapy, particularly for conditions like Leber congenital amaurosis, has already received clinical approval, and many additional gene therapies are under active development. Furthermore, retinal pigment epithelium and stem cell transplantation are progressing, with several phase I/II trials ongoing. Many other promising therapies are also being explored.

However, these approaches are limited to retinal interventions and cannot address conditions like retinal detachment or diseases causing irreversible damage to retinal ganglion cells or the optic nerve, such as congenital childhood glaucoma. This limitation has driven significant interest in vision restoration by directly targeting the brain’s cortical visual centers. Since 2017, three clinical trials for visual cortical prosthetics have been initiated. These trials, grounded in decades of research on cortical stimulation (see Table 1), have examined both short-term and long-term effects. However, the findings so far are primarily descriptive.

▷ Table 1: Studies describing perceptual effects of human cortical electrical stimulation.

Predicting the level of vision restoration achievable with visual cortical prosthetics before human implantation remains a significant challenge. The field of neural implants continues to rely heavily on experience and assumptions. For example, it seems logical that increasing the number of smaller electrodes would enhance resolution. But is this assumption valid?

A team led by Ione Fine 14 developed a computational “virtual patient” model based on the neurophysiological structure of the primary visual cortex (V1). This model accurately predicted perceptual outcomes from electrical stimulation, including the position, size, brightness, and spatiotemporal shape of elicited visual perceptions. Such predictions provide a valuable tool for anticipating the perceptual effects of visual cortical prosthetic implants before their application in humans.
 

2. Addressing Data Shortage with a “Virtual Patient” Model

Building on previous models of retinal prosthetic stimulation, Ione Fine’s team has developed an innovative theoretical framework. The model integrates currents from cortical implants over time, converting them into neural signal intensities. The resulting perception from neuronal stimulation is based on a linear summation of each cell’s receptive field, adjusted by the neural signal intensity at each specific location and moment. Despite its mathematical simplicity and lack of complex parameters, the model accurately predicts a range of data on cortical stimulation.

▷ Fine, Ione, and Geoffrey M. Boynton. “A virtual patient simulation modeling the neural and perceptual effects of human visual cortical stimulation, from pulse trains to percepts.” Scientific Reports 14.1 (2024): 17400.

(1) Temporal Conversion from Pulse Trains to Perceptual Intensity

To design the model, Ione Fine’s team first employed a rapid temporal integration phase, capturing the immediate response of cells to current as a measure of “spike activity intensity.” They also introduced a refractory period, positing that cells require a brief interval after activation before they can respond to subsequent stimuli. This phase is followed by a slower integration process, incorporating a compressive non-linear relationship, as shown in Figure 1.

▷ Figure 1. Schematic of the conversion from pulse trains to perceptual intensity over time

The team implemented this process using a straightforward single-stage leaky integrator. In this model, the depolarization rate depends on both the current depolarization level and the input current, meaning that variations in current directly influence the rate of depolarization. In the model’s first phase, the “spike response intensity” reflects the collective recruitment of spikes from various cell populations with differing activation thresholds. The compressive non-linear function not only accounts for saturation effects in these populations but also mirrors complex cortical gain control mechanisms.

(2) Visual Dominant Columns, Orientation Pinwheels, and Receptive Fields

The primary visual cortex (V1) contains complex, structured neural arrangements that influence how visual information is perceived and processed. Building on the work of Rojer and Schwartz, Figure 2 illustrates these simulated structures. The orientation columns (Figure 2B), sensitive to visual direction, were generated by band-pass filtering random white noise to reflect neurons’ directional preferences. Fine’s team further enhanced the model by incorporating visual dominant columns (Figure 2C), which add directional gradients to the same noise signal, resulting in orthogonally arranged visual and orientation columns. These structures closely mimic maps observed in both macaques and humans.

A single receptive field (Figure 2F) is the basic unit in the visual system responsible for receiving and processing light signals. It is generated using a simple model through the additive combination of “ON” and “OFF” subunits, with the spatial separation of subunits derived from a unimodal distribution. The same band-pass filtered white noise used to generate orientation and visual dominant maps was also used to create controls for the separation of “ON” and “OFF” receptive fields (δ_on–off) and the relative strength of “ON” and “OFF” receptive fields (w_on–off).

Predicted phosphenes are produced by summing receptive field contours at each cortical location, with intensity weighted according to the level of electrical stimulation. Consequently, the stimulation intensity at an electrode directly influences the perceived phosphene’s brightness and size.

▷ Figure 2. Schematic of the cortical model (A) Transformation from visual space to cortical surface. (B) Orientation pinwheel map. (C) Visual dominant columns. (D) Spatial separation of ON and OFF subunits. (E) Relative strength of ON and OFF. (F) Receptive field size.

(3) Phosphene Thresholds and Brightness as Functions of Electrical Stimulation Temporal Characteristics

Ione Fine’s team compared the model’s predictions with data on current amplitude thresholds and brightness ratings measured across various pulse sequences. The model accurately described how pulse sequences are converted into perceptual intensity, successfully predicting phosphene thresholds and brightness ratings across different pulse parameters, electrode locations, and electrode sizes. This suggests that the model can reliably estimate patients’ perceptual experiences regardless of changes in stimulation frequency, pulse width, or the specific location and size of electrodes.

*Phosphene threshold refers to the minimum current intensity required to produce a visible phosphene, while brightness rating is patients’ subjective evaluation of the phosphene’s brightness.

(4) Relationship Between Phosphene Size, Current Amplitude, and Eccentricity

Additionally, the model developed by Ione Fine’s team 1 successfully predicted how phosphene size varies with current amplitude and visual field eccentricity. The study found that as current amplitude increases, phosphene size also increases, closely aligning with patient perception data. This indicates that current amplitude is a key factor in determining phosphene size. Furthermore, the model demonstrated that phosphene size grows with visual field eccentricity, meaning that electrical stimulation at the periphery produces larger phosphenes than those in the central area.

(5) Shape Recognition

In the study, the team compared perceptual experiences generated by simultaneous versus sequential electrode stimulation. The results showed that when multiple electrodes were stimulated simultaneously, the model failed to correctly recognize complete letter shapes, mirroring patients’ actual experiences. However, when electrodes were stimulated sequentially in the order of writing, the model accurately recognized letter shapes. This highlights a critical point: simultaneous stimulation makes it difficult to group and interpret phosphenes correctly, whereas sequential stimulation facilitates the formation of recognizable shapes.

This phenomenon may arise from the absence of the Gestalt effect in the model. Gestalt psychology posits that the whole is greater than the sum of its parts, meaning our perceptual system integrates dispersed phosphenes into meaningful wholes. However, since the model does not account for electric fields or complex neural spatiotemporal interactions, phosphenes cannot be effectively grouped during simultaneous stimulation, leading to challenges in shape recognition. Sequential stimulation, by temporally separating stimuli, reduces interference between phosphenes and allows the perceptual system to better integrate information and recognize shapes accurately.

(6) Using “Virtual Patients” to Predict Perceptual Outcomes of New Devices

The ability of Ione Fine’s team’s model to replicate extensive data suggests it can provide insights into the potential perceptual experiences of new technologies—one of the primary uses of the “virtual patient” model.

Figure 3 examines the perceptual experiences produced by different electrode sizes using the “virtual patient” model. In Figure 3A, the simulated array employs extremely small electrodes (tip areas ranging between 500-2000 μm²). The model predicts visual phosphenes that align with preliminary experimental data: phosphenes from adjacent electrodes cannot be clearly distinguished. This matches patients’ pre-experimental observations, indicating that when electrode stimulation intervals range from 0.4 to 1.85 millimeters, both single and multiple electrode stimulations produce irregularly shaped phosphenes. This implies that extremely small electrodes may simultaneously stimulate neuronal populations tuned to similar directions, resulting in elongated or complex phosphene structures.

Figures 3B and 3C further explore the impact of electrode size on patient perception. For small electrodes with limited current diffusion (cortical tissue stimulation radius less than 0.25 millimeters), the model predicts complex phosphene structures, as shown in the upper part of Figure 3B. At this stage, electrode size has minimal impact on phosphene appearance or size. When the electrode radius ranges from 0.25 millimeters to 1 millimeter, phosphenes begin to resemble “Gaussian blobs,” but their size is still primarily determined by receptive field size rather than the stimulated area. Only when the electrode radius exceeds 1 millimeter does electrode size significantly influence phosphene size.

Crucially, the model indicates that throughout the visual field, receptive fields impose a physiological “lower limit” on phosphene size. Specifically, reducing the stimulation area’s radius below 0.5 millimeters may not significantly enhance vision and might instead render phosphenes difficult to interpret.

▷ Figure 3. Using “Virtual Patients” to Predict Perceptual Outcomes
(A) Simulated perception with very small deep electrode arrays. The lower left panel shows electrode positions and sizes in the array. The upper right panel displays example perceptions from three individual electrodes. The lower panel shows predicted outcomes when electrode pairs are stimulated simultaneously. (B, C) Simulated predicted phosphene shapes and sizes under different electrode sizes and cortical locations. The narrow shaded area in panel C represents the 5–95% confidence interval.

Similarly, the question arises: does having more electrodes lead to better vision restoration?

Ione Fine’s team simulated perceptual outcomes for three different electrode array configurations through Figure 4, revealing the complexity of this issue.

Figure 4A shows electrodes arranged in a regular pattern within the visual space. This arrangement produces sparse, small phosphenes in the foveal region (the retina area responsible for high-resolution vision), significantly underestimating its perceptual capability. This means that despite an increased number of electrodes, the phosphene distribution in the foveal region is not dense enough to fully utilize its high-resolution potential.

▷ Figure 4. Comparison of Simulated Electrode Array Configurations
(A) Regularly spaced electrodes in the visual field. (B) Regularly spaced electrodes on the cortical surface. (C) ‘Optimal’ spacing.

Figure 4B shows electrodes arranged regularly on the cortical surface. This configuration leads to an excessive concentration of electrodes in the foveal region, causing significant overlap between receptive fields. Consequently, these overlapping receptive fields do not substantially improve resolution. In fact, electrodes near the foveal region almost project to the same visual spatial location, making it difficult for patients to perceive positional changes even with slight electrode movements. This finding challenges the traditional view that the extensive expansion of the foveal region in V1 supports higher spatial sampling capabilities.

Figure 4C presents an “optimal” electrode configuration, where electrode spacing is designed so that the center-to-center distance of stimulated phosphenes maintains a fixed ratio with phosphene size. Since receptive field size increases linearly with visual field eccentricity and cortical magnification changes logarithmically, this configuration results in a more dispersed electrode distribution in the foveal region compared to peripheral areas. The results indicate that electrodes should be more sparsely arranged in the foveal region rather than densely packed, contrary to common intuition.

The simulation results from Ione Fine’s team demonstrate that excessively dense electrode arrangements in the foveal region do not enhance perceptual resolution. Instead, based on neurophysiological constraints, appropriately distributing electrode positions and spacing is essential to maximize the perceptual efficacy of visual cortical prosthetics.
 

3. Insights from the “Virtual Patient” Model

The “virtual patient” model tackles a fundamental challenge in vision restoration: predicting patients’ perceptual experiences before device implantation. This model enables researchers to evaluate how different electrode designs and stimulation parameters might impact visual perception, all without actual implantation.

Additionally, the model challenges a common misconception: increasing electrode numbers or reducing their size does not necessarily improve visual perception and may actually complicate perceptual experiences.

Ione Fine’s model identifies three main factors limiting the spatial resolution of cortical implants: cortical magnification, receptive field structure, and electrode size. Receptive field size is strongly linked to cortical magnification. In most cortical areas, the receptive field area is roughly to the -2/3 power of cortical magnification. This relationship supports prior findings by Bosking and colleagues, who noted that patient-drawn phosphene sizes could be predicted by cortical magnification. In the foveal region, where cortical magnification peaks, receptive fields are smallest, with radii ranging from 0.02 to 0.5 degrees.

Data and simulations show that for a fixed electrode size, phosphene size increases linearly with eccentricity. When electrode radii are smaller than 0.25 millimeters, this relationship primarily reflects the increase in receptive field size with eccentricity. For larger electrodes, both cortical magnification and the size of the stimulated cortical area become significant factors in determining phosphene size.

With smaller electrodes, receptive field size primarily limits visual acuity. If neurons with very small receptive fields could be selectively stimulated, closer electrode spacing in the foveal region could enhance spatial resolution. However, human vision can resolve details far finer than a single receptive field width. This fine discrimination depends on interpreting complex response patterns across neuron populations with diverse receptive fields, rather than on individual neurons alone.

In summary, Ione Fine’s simulations suggest that, in the near future, the spatial resolution of visual cortical prosthetics will likely be constrained more by the neurophysiological structure of the visual cortex than by engineering limits. This implies that enhancing vision restoration outcomes will require a deeper understanding and application of visual cortex neurophysiology, rather than simply increasing electrode numbers or reducing their size.
 

4. Current Limitations of the “Virtual Patient” Model

The advent of the “virtual patient” model has transformed our understanding of retinal implant surgeries. While modeling techniques have long simulated the effects of electrical stimulation on local tissues, such as current diffusion from electrodes, predicting perceptual outcomes requires expanding virtual models to include core physiological principles.

The “virtual patient” model developed by Ione Fine’s team has effectively predicted a range of perceptual outcomes from cortical electrical stimulation, suggesting it may serve as a useful approximation for future retinal or cortical implants. This model could guide future research using more reliable subject data.

Despite its utility, the current model has several limitations. First, it uses current amplitude as an input parameter, though a more precise approach would employ current density (current intensity divided by electrode area) to better capture current distribution and its effects within tissue.

Second, the model assumes that electrodes are perfectly aligned with the cortical surface. In reality, electrodes may not sit flush, and even slight tilts can result in only the edges driving neural responses effectively.

Third, the model serves only as an approximation and may not be suitable for long-term stimulation protocols. Fourth, it excludes electric fields and nonlinear neural interactions. Fifth, it assumes perception is a simple average across receptive fields. An alternative could involve representing each neuron by its “optimal reconstruction filter”—the cell’s specific role in reconstructing natural images within the neural network.

Finally, the current model only includes the V1 cortical area. Due to the cortical surface’s configuration, electrode implantation is much easier at higher-level visual areas, such as V2 or V3. Many components of the model, including the transformation from visual space to the cortical surface, could be easily extended to these higher visual areas. The model could also be expanded to incorporate the receptive fields of V2 or V3 neurons. However, the complexity of V2-V3 receptive field structures, combined with a lack of cortical stimulation data from V2 or V3 electrodes, means that any such extension of the model remains highly speculative at present.

5. Future Outlook

In the future, “virtual patient” models may serve multiple critical functions. For researchers and companies, they can quantitatively evaluate our level of understanding of this technology. Given the difficulties in collecting behavioral and cortical data, model-driven approaches can help guide experiments to produce the most meaningful insights, thereby optimizing the allocation of research resources.

Another important application is predicting the visual quality a specific implant might provide. In this study, Ione Fine’s team used qualitative assessments of perceptual quality to evaluate different array configurations. A more rigorous approach might involve subjective interpretation; for example, asking visually normal individuals to perform perception tasks with simulated prosthetic vision. Alternatively, simulations could serve as inputs for decoders trained to reconstruct the original image. Recent cortical simulators display characteristics similar to those in more complex models.

Finally, “virtual patients” can guide the development of new technologies. The current model, for example, suggests that small electrode sizes and dense implantation in the foveal region provide limited advantages. Additionally, “virtual patients” could help generate optimized training sets for deep learning-based prosthetic vision, aiding in the identification of optimal stimulation patterns for existing implants. Similar retinal stimulation models are currently used to simulate and enhance prosthetic vision in virtual reality environments by generating preprocessed training data for deep learning.

For agencies like the FDA and insurance providers, these models can inform essential visual tests for device evaluation, helping to establish scientific standards that ensure the safety and effectiveness of new vision restoration technologies. Finally, for surgeons and patients’ families, these models provide more realistic expectations for perceptual outcomes.