### Introduction

- Top of page
- Abstract
- Introduction
- Box 1 Fourier description: amplitude, phase, power
- Combined space and spatial-frequency representations
- Hyperacuity
- Non-Fourier Kernel functions
- Acknowledgements
- References
- Appendix
- Supporting Information

When describing visual targets and the changes they undergo in transmission through the eye’s optics to become retinal images and then in their further processing by the retinal and neural stages of vision, the time-honored way is to depict the light intensity or excitation level in transverse planes. A more recent alternate is the spatial frequency or Fourier transform representation, Equation 1. Provided that linearity constraints are met and that the phase of the Fourier components is not neglected, the two are equivalent.^{1} Because spatial frequency and spatial extent are reciprocal measures, they tend to emphasize different aspects of any given situation. Feature size and shape are seen best in the spatial coordinates, whereas the presence and resolution of fine details are emphasized in the high spatial-frequency Fourier components.

- (1)

Equation 1 Fourier transform of I(x) the light distribution in one dimension in a lateral plane, giving the distribution of amplitude I(n) and phase fn of the cosinusoidal patterns of spatial frequency n that describe the light distribution equivalently.

In *Figure 1* there are four examples of targets, a sinusoidal grating, a wide bar, a line, and finally, a line pair whose separation can be discriminated with hyperacuity precision.^{2} Illustrated are the frontal view of the patterns in their lateral planes normal to the optical axis, the light distribution *I*(*x*) along the *x* axis in the middle column, and *I*(*ν*), the Fourier counterparts of *I*(*x*), where amplitudes alone are given because, for simplicity’s sake, targets have been selected for which phase does not enter. The emphasize the salient aspects, some simplifying assumptions have been made: the grating in the first row and the black background in the other rows is assumed to extend to infinity, and the lines in row *D* are assumed very narrow.

If the entire light distribution along the *x* axis is used as the basis for Fourier transformation, it suffices to plot the spatial frequency representation along a single line orthogonal to the *x* axis, but if phase is involved, this will not be just a vertical ordinate at points along this *ν* axis, but a vector with amplitude and phase, represented by the orientation angle in the plane parallel to the *ν = *0 plane. Even if the pattern is symmetrical, phase changes will be introduced if the analysis is not centered on an axis of symmetry. For example, if the line in row *C* of *Figure 1* is shifted laterally, the amplitudes of the vectors at all *v* values will not change. But because any lateral shift in *x*-distance becomes an increasing phase angle with increasing spatial frequency of the cosinusoidal components, the phase angles increase sequentially. This is illustrated in *Figure 2*, which shows the spatial-frequency locus for the thin line when it has been laterally shifted by 6 arcmin. The phase, whose angle is proportional to the frequency, results in the spiral appearance of a three-dimensional amplitude/phase/spatial-frequency representation.

It follows that the power spectrum, based on only the length of the vectors from the *ν* axis to each point on the spatial-frequency locus, thus omitting phase information, is an incomplete representation. The actual target cannot be reconstructed from it.^{3} It has been demonstrated that natural images have mostly similar power spectra.^{4} Since for survival we depend on recognizing differences in natural scenes, and these differences reside in the phase information, the study of spatial vision necessarily has to transcend stimulus power spectra.

### Box 1 Fourier description: amplitude, phase, power

- Top of page
- Abstract
- Introduction
- Box 1 Fourier description: amplitude, phase, power
- Combined space and spatial-frequency representations
- Hyperacuity
- Non-Fourier Kernel functions
- Acknowledgements
- References
- Appendix
- Supporting Information

For *I*(*x*), a distribution in the spatial dimension *x,* the related distribution *F*(*v*) along *v,* the spatial frequency dimension, is given by

For functions with even symmetry, i.e. for which the conditions *I*(*x*)* = I*(*−x*) holds, the *sin* components vanish and *F*(*ν*)* = *∫*I*(*x*) (*cos*(2*πxv*)*dx.* Otherwise the elements at the spatial frequency *v* of *F*(*v*), the transform of *I*(*x*), can be regarded as a vectors of amplitude *A*_{v}* = √*(*C*(*v*)^{2 }*+*^{ }*S*(*v*)^{2}) and phase angle *φ** =* *tan*^{−1} (*S*(*v*)*/C*(*v*)).

The meaning of this statement is the following. To reconstitute the original function *I*(*x*) from a knowledge of *F*(*v*), for each frequency *v* the functions *cos*(2*πνx*) and *sin*(*2πνx*) are multiplied by *C*(*v*) and *S*(*v*) respectively and products summed. Alternatively, to reconstitute the original function *I*(*x*) from the distribution of amplitudes *A* and phases *φ*_{ν,} the contributions *A*_{v}*cos*(*2πνx + φ*) are summed over all values of *ν.*

The power spectrum is given by

### Combined space and spatial-frequency representations

- Top of page
- Abstract
- Introduction
- Box 1 Fourier description: amplitude, phase, power
- Combined space and spatial-frequency representations
- Hyperacuity
- Non-Fourier Kernel functions
- Acknowledgements
- References
- Appendix
- Supporting Information

Not so well-known is a mode of representation which combines both the space and spatial-frequency domains. It is more easily studied by confining consideration to a single lateral image dimension which now becomes one of the axes of a two-dimensional distribution, the other, spatial frequency along the *ν* axis, being drawn orthogonally. This suffices for light patterns with even symmetry for which all phases are zero. An example of the concept is provided by *Figure 3*, a 3-dimensional configuration with the light distribution (here a one-dimensional Gabor blob) drawn along the *x*-axis and its spatial frequency spectrum along the orthogonal *ν* axis.

Often there is an advantage to concentrate on the frequency content not of the whole of the distribution but of those segments in which there are rapid changes, as for example in the vicinity of borders. Analysis of acoustic signals is an area where this problem is acute. Though the longer the segment of a sound recording that is being subjected to probing, the more it satisfies the criteria of strict Fourier theory, what usually prompts study are local differences in the temporal stream, as one musical note following another, or the vowels preceding consonants.

Restricting the signal train that is being subjected to frequency analysis is best managed not by chopping it into segments but rather by multiplying it with an overlaid windowing function and performing a Fourier analysis over the narrowed regions of the test distribution. The operation is repeated, shifting the windowing function over the test distribution sequentially point by point. This yields a picture of the, so to speak, ‘local’ Fourier content of the distribution being analyzed and it is linked to its source by being graphed orthogonally at each position of the window, resulting in a surface.

The formula used in this approach is given in Equation 2, where *g*(*x*) is the windowing function extending over a range no wider than the source distribution and usually much less. At each point *x* one computes a temporary subsidiary distribution, consisting of a segment of the full distribution limited to the segment *x ± t*_{m} multiplied by the windowing function, and performs a Fourier analysis over this temporary subsidiary contribution. The result is plotted along the *ν* axis orthogonal to the distribution’s axis at the point *x,* giving a surface when carried out all along the source distribution. Of specific interest is the height along the *ν =* 0 line. Then the cosine factor in Equation 1 is 1.00, leaving, as the remaining computation, the sum of the product of the heights of the distribution and of the windowing function centered on each point *x* over the region from *x−t*_{m} to *x+t*_{m}.

- (2)

Equation 2 Generic equation for generating the two-dimensional function G(x,n), which has the convolution of the target distribution I(x) with a windowing function g(x) along the x axis, and the Fourier transformation of this convolution over the limited domain x + tm along the n axis.

What has been achieved is a 2-dimensional representation, simultaneous in the space and spatial frequency domains, of two aspects of a single object or image dimension. Plotted along the *ν = *0 line is no longer the original distribution but its convolution with the particular windowing function that is required to reach a compromise between one so narrow as to be meaningless for Fourier purposes and one so wide that significant local differentiations within the test distribution have been washed out. For that reason in some of the following graphs the originating distribution has been superimposed, as a reminder of what had been started with. Nothing changes in principle for extension from simple symmetrical distributions to cases when the phase matters and each point in the *x,ν* plane has as its *z* coordinate not just an amplitude but a complex number representing amplitude and phase.

Some widely used exemplars of the windowing function can be cited. The first is the one introduced by Gabor, in which *g*(*x*) is the Gaussian function exp(*−*(*xk*)^{2}). Here *k* is free parameter, an inverse measure of the width, which must be specified; *g(x)* falls to 0.02 at ± 2/*k* and thus governs the range of integration *x *± *t*_{m}. Gabor transforms of three of the targets in *Figure 1* are shown in *Figure 4*.

The concept of windowing functions for purposes of limiting the region within which Fourier transformation takes place, can be extended into the field of visual processing but, as soon as neural components enter, linearity can no longer be taken for granted and rigor may be compromised. Nevertheless the approach has value in allowing visualization of the changes that signals undergo in their passage from object to retinal image and then subsequently to the sensorium. The eye’s point-spread function, these days capable of good specification by means of wavefront technology, is unobjectionable on this count, but more elaborate excitation spread functions, which might include both optical light spread in the eye media and retinal center/surround antagonism, need to be treated with care. To model the center/surround antagonism of retinal functioning, a difference of two Gaussian (DoG) windowing function can be employed. The resulting distributions (*Figure 5*) highlight the band-pass operation thus performed. This process has been utilized in, for example, deconstructing the geometrical-optical illusion known as the shifted-chessboard or café-wall patterns.^{5}

A different approach is taken in the so-called Wigner distribution function (Equation 3), originating in theoretical physics to define simultaneously the position and momentum of a particle. It has found traction in modern optics^{6,7} where its manifest orthogonality matches

- (3)

Equation 3 for the Wigner distribution function, W(x,n) is a specific implementation of the general transformation distribution in Equation 2 in which the windowing process is accomplished by means of the autocorrelation of the original target distribution I(x). The domain used for the Fourier transformation is now the whole distribution.

that inherent in a function and its Fourier transform, as realized in the relationship between the distribution of coherent light in the transverse planes containing a distant object and the system’s aperture. The spatial basis for Fourier transform computation now is the distribution’s autocorrelation, and this has the virtue of having the distribution itself govern the windowing range instead of it being imposed by the investigator, as is the case with the Gabor and DoG transforms. The Wigner distribution function has some special properties: Its integral gives the total energy that is being transferred and it undergoes defined transformations in the process of free-space propagation and imaging by lenses.^{6} Wigner distribution functions for the three of the four targets in *Figure 1* are shown in *Figure 6*.

For practical purposes, these representations allows the opportunity to survey the spatial frequency contents at points on the *x* axis, and thus to gauge the resolution needed for reconstruction of specific regions of the target.

Another appropriate windowing function is the cross-sectional light distribution of the Airy disk, (*J*_{1}(*x*)*/x*)^{2}, which defines the effect purely of diffraction in optical imagery in monochromatic light of wavelength *λ* and a circular pupil of diameter *a*. Its Fourier transform is the optical transfer function.^{1} When the computation is performed with sufficient rigor, it will demonstrate that under these circumstances there is a cut-off spatial frequency *v = a/λ* beyond which all values in the *x,ν* plane are zero.

A related but somewhat different approach is the wavelet methodology,^{8} which does not feature a single windowing function, like the Gaussian in the Gabor and DoG transforms, or the one that is determined by the distribution itself in the Wigner function, but one whose width varies inversely with the Fourier frequency. Though this makes the computational basis uniform across the frequency spectrum, by the same token it fails to concentrate singularly on specific locations within the distribution.

These distributions have the interesting property that whereas in a given situation the total footprint remains the same, their shape can vary.^{9} A specific example is pure magnification, were any stretch in the *x* dimension is accompanied by concomitant reciprocal contraction of the spatial frequency scale. The mode of analysis thus can help in the design of display situations. Aperture and wavelength determine the maximum spatial frequency transmission of the system. The magnification can then be adjusted so that the target’s spatial frequency demands are met by the instruments’s optical transfer capabilities, necessarily of course at the expense of target extent.

### Hyperacuity

- Top of page
- Abstract
- Introduction
- Box 1 Fourier description: amplitude, phase, power
- Combined space and spatial-frequency representations
- Hyperacuity
- Non-Fourier Kernel functions
- Acknowledgements
- References
- Appendix
- Supporting Information

To illustrate many of the concepts that have been introduced, an example will be drawn from the topic of visual hyperacuity, the normal human observer’s ability to make spatial localizations with precision better, by almost an order of magnitude, than the eye’s resolution limit and the image binning imposed by retinal receptor packing. Vernier alignment thresholds of a few arcsecs are an example of what is involved, but for purposes of this discussion a related threshold will be used, the discrimination of the separation of two edges or lines. Most observers can distinguish between line pairs 4 arcmin and 3.9 arcmin apart, whereas the best optical resolution thresholds, and also the diameter of foveal cones, are between 0.5 and 1.0 arcmin. These targets, quite typical of hyperacuity patterns, have the advantage that, unlike misaligned vernier stimuli, their full description and analysis can be carried out in a single dimension and that, because they have even symmetry, the phase of their Fourier components can be ignored, i.e. purely cosine transformation suffices.^{10}

The basic conformation of the line-pair pattern in the space, spatial frequency, Gabor, DoG and Wigner transformations are shown in *Figure 7*. The just distinguishably-different pattern, 3.9 arcmin apart, will have their scales differ from the above by 2 ½%, diminished in the space and augmented in the Fourier domains. A standard deviation of 1 arcmin has been chosen for the Gabor transform, a value that is representative of the image spread, and the values for the DoG adequately model space constants for foveal visual processing.

Each display shows a different facet of the situation. The Fourier transformation (*Figure 7b*) covers the entire pattern and can also serve as a reminder of the two slits/sinusoidal fringe space/spatial frequency duality that is familiar in Young’s interference fringe experiment of wave optics. The Gabor transform (*Figure 7c*), which might be an approximate representation of the energy in the optical retinal image or perhaps the pattern of retinal receptor excitation, highlights the spatial regions in which processing places the most demand on higher spatial frequencies. It allows, for example, the gauging of the effect on the pattern of curtailing the spatial-frequency spectrum by, e.g. pupil size reduction or optical blur, something that would not be obvious by just looking at the other panels. The DoG transform (*Figure 7d*) is a representation of the activity of an array of retinal ganglion cells which has the center/surround antagonism incorporated.^{5} Finally, the Wigner distribution function (*Figure 7e*), is a richer source of information than the others. The strips along the *ν* axis intersecting the *x* axis at ± 2 arcmin. The location of the two target lines, are closely related to what is contained in the other transforms, but in addition there is a great deal of activity in the center. The distribution along the *ν* axis at *x = *0 is actually equivalent to what is shown in *Figure 7b* and constitutes the spatial frequency demands in the sensorium as a whole of the double-line target. If image analysis were entirely in the Fourier domain, something that is not the case in human vision, this is where the emphasis would be.

In a hyperacuity determination, the observer makes the distinction between, say, 4′ and 3′54″ line-pairs that are presented not superimposed but individually, either side by side or one after the other. Because it matters in what order the differencing and transform operations are performed – the difference between the Wigner transforms of two signals is not equal to the Wigner transform of the difference between the signals – it needs consideration which of the two sequences might be involved in neural processing. By the nature of the task, an instantaneous difference signal is an unlikely event. More probable the representations of the two targets are examined separately and then compared. The neural apparatus used for such discrimination is as yet unclear. *Figure 8*, containing an assembly of the differences in the targets, and in their Fourier, Gabor, DoG and Wigner transforms highlights the location within these distributions where differences are most pronounced. The magnitude of the effects, as compared with the data in *Figure 7*, is much less.

The small values of hyperacuity thresholds raise the question of how they relate to the ultimate limits of information transfer through optical devices mandated by diffraction theory. The minimum spread of a point object into Airy disks, whose diameter depends directly on the wavelength and inversely on the system aperture, enables visualization of the resolution difficulties as they are encountered when such images of two closely-adjacent objects overlap. But a more succinct expression is embodied in the cut-off spatial frequency which specifies the spatial frequency band that can be admitted to form an image. For wavelength *λ* and aperture *a* (in the same units, say, *mm*) it is *a/λ* cycles/radian visual angle at the entrance pupil. In spite of much current writing on superresolution,^{7,11,12} none of the phenomena subsumed under this term actually ‘break the diffraction limit’ if by that is meant that they violate the Uncertainty Principle of quantum theory, which limits the simultaneous measurement of position and momentum of elementary particles – here photons whose direction of propagation becomes uncertain as the width of the aperture through which they pass is restricted. In superresolution as currently understood – of which there are many examples, starting early-on in microscopy with dark-field illumination – target details beyond the conventional diffraction limit are passed through optical system by exchanging or multiplexing spatial-frequency bandwidth. In the same vein, no hyperacuity phenomena ‘break the diffraction limit;’ rather they optimize information provided within it, usually in a way that might be described in terms of Bayesian inference. Distinctions are made on the basis of differences *within* the cut-off spatial frequency and, by virtue of prior understanding, on the tacit assumption that nothing beyond it enters in the decision.

The basic approach can be judged from *Figure 7e*. Because the target is spatially delimited, the Wigner distribution function of the 2-line target pattern can be extended arbitrarily far into the Fourier domain. However, in actuality it would be truncated at the cut-off spatial frequency of the eye’s optics, <60 cycles per degree in the natural state. In practice this means what is available to the observer contains an uncertainty of what may have been beyond that truncation line. However, in any hyperacuity test – and usually in equivalent practical situations as well – prior constraints have been set on the range of possible targets; the decision involves the image structure, the likelihoods that it arose from that range of physical targets and the probabilities that such targets had actually been shown.

### Non-Fourier Kernel functions

- Top of page
- Abstract
- Introduction
- Box 1 Fourier description: amplitude, phase, power
- Combined space and spatial-frequency representations
- Hyperacuity
- Non-Fourier Kernel functions
- Acknowledgements
- References
- Appendix
- Supporting Information

Equation 1 is merely a special case of the more general class of kernel analysis in which the target distribution is being sifted by cosines of a range of frequencies to yield a description in terms of their amplitude and phases. Functions other than the trigonometrical ones that are the basis for the Fourier approach can be used but mostly they lack the advantage of permitting a reversal of the process that allows a unique reconstruction of the source distribution from the transformation. The prominence of the Fourier method is rooted in the firm foundation it has in diffraction theory, which remains an obligatory ingredient in optical imagery. When it comes to visual sensory mechanism, however, whose beginning processing is essential local, it is attractive to consider the use of spatially more restricted kernels. Mathematically, this is simply expressed by substituting for the cosine term in Equation 1 some other function whose parameter can also take a range of values, equivalent to spatial frequency and also with increasing spatial compactness with increasing values. Two specific examples are the Gaussian and the Gaussian derivative, illustrated in *Figure 9a* and *b*, respectively, where width is governed by a single parameter, reciprocally related to spread.

The difference between the Gabor transform of Equation 2 and the distribution resulting from sifting a target light pattern through a series of Gaussian kernels is important. The former results when the target is sifted by a *single* Gaussian and then subjected to a Fourier analysis. What is being studied in this section, on the other hand, has nothing to do with sinusoids. It is the output when a target distribution is analyzed at each of its points through a series of Gaussians of gradually diminishing widths.

This procedure, using the Gaussian derivate function 9B, has been performed for a visual field, half of which is uniformly illuminated, the other half dark: a light step. The graph in *Figure 10* can be interpreted as the level of stimulation that might be experienced at each location *x* by a hypothetical array of edge detectors with a range of accepting functions with decreasing widths and shows those that are most affected in the various locations along the target’s *x* dimension. Based on empirical evidence showing segregation into on and off streams^{13} and separate processing of edges with opposing polarity,^{14,15} the functions illustrated in *Figure 9* should each be considered in two flavors, the Gaussian also going negative, and the Gaussian derivative also with reversed edge polarity.

The graphical depiction in *Figure 10* provides an accessible and easily visualized framework for some size-tuned sensory mechanisms that have been proposed over the years.^{16,17} Most such formulations assume that the distribution of detector widths is not continuous, but concentrated into clusters, called channels. Such a discrete tiling of the acceptance zone would be accommodated by compartments in the *x–k* plane of the manifold in *Figure 10* presumably with some overlap and changes in scale from fovea into the retinal periphery. The sensory response to a given target would then be determined by how well the stimulus distribution is matched by the topography of the detection apparatus.