Simultaneous multi‐spectral, single‐photon fluorescence imaging using a plasmonic colour filter array

We present the first realisation of simultaneous multi‐spectral fluorescence imaging using a single‐photon avalanche diode (SPAD) array, where the spectral unmixing is facilitated by a plasmonic metasurface mosaic colour filter array (CFA). A 64 × 64 pixel format silicon SPAD array is used to record widefield fluorescence and brightfield data from four biological samples. A plasmonic metasurface composed of an arrangement of circular and elliptical nanoholes etched into an aluminium thin film deposited on a glass substrate provides the high transmission efficiency CFA, enabling a bespoke spectral unmixing algorithm to reconstruct high fidelity, full colour images from as few as ∼3 photons per pixel. This approach points the way toward real‐time, single‐photon sensitive multi‐spectral fluorescence imaging. Furthermore, this is possible without additional bulky components such as a filter wheel, prism or diffraction grating, nor the need for multiple sample exposures or multiple detectors.


| INTRODUCTION
Fluorescence imaging is a common approach used to investigate biological cells and compounds, and is used in medical research and diagnosis [1], botany [2], and human [3] and animal [4] biology. Certain organic compounds within samples may exhibit endogenous fluorescence; however, for many applications a chemical fluorophore is used to label the cellular or molecular sample under study. This can allow for highly-specific labelling of target proteins or structures of interest, with the fluorophores emitting light of specific wavelengths upon excitation of the sample. In widefield systems the emitted photons are then collected by a camera, typically either an intensified CCD (ICCD) [1,3] or an electron bombarded CCD (EBCCD) [5]. Scientific CMOS (sCMOS) sensors are also used in combination with an image intensifier [6][7][8], which offer frame rate advantages over CCD-based detectors, but these sensors generally exhibit comparatively higher noise levels and are therefore unable to offer the single-photon sensitivity of ICCDs or EBCCDs, which may be required for low photon-flux applications. Electron-multiplying CCDs (EMCCDs) are also fairly common in fluorescence microscopy [9,10]. They can be of equivalent sensitivity to ICCDs or EBCCDs; however, the internal amplification process means the devices generally require Peltier cooling to keep thermal noise levels down. For confocal and other scanning-based systems, single-pixel photomultiplier tube (PMT) detectors are most commonly used [11][12][13], but these are difficult to scale up into large-format arrays for widefield imaging scenarios.
Recently, the development of large pixel format singlephoton avalanche diode (SPAD) arrays has led to their use in biophotonics applications due to their low-cost and frame-rate advantages. While the particular detector array used in this work does not contain the timing electronics necessary for lifetime measurements, in-pixel timing circuitry in some SPAD arrays provides time-correlated single-photon counting (TCSPC) capabilities making them increasingly attractive propositions for widefield fluorescence lifetime imaging microscopy (FLIM) [14][15][16][17][18]. Singlepixel SPAD detectors are also finding use in scanning systems for fluorescence measurements [19] which can provide very high efficiency; however, these systems are comparatively slow and incompatible with imaging of dynamic processes. EMCCDs cannot be gated and are therefore unsuitable for timing applications; however, ICCD, EBCCD and sCMOS cameras can. CCD and sCMOS based cameras, however, cannot offer the picosecond temporal resolution possible with TCSPC-based SPAD detector arrays (10 − 100 ps [20]), with their reliance on the sliding-gate technique to extract timing information, where a series of time-gated exposures are collected with an iteratively increasing delay to the sensor timing gate [21][22][23][24]. Two hundred picosecond represents the current minimum gate duration available for an ICCD [25], with a gate rise-time facilitating a 10 ps timing resolution; however, a >100 ps timing resolution is more common. Furthermore, this method generally requires many exposures to build up an accurate timing histogram, although novel single-shot approaches have been made [26]. Frequencydomain microscopy approaches have also been employed to achieve similar timing resolution in FLIM measurements (100 ps) [27,28]. The use of SPAD arrays on a widefield fluorescence imaging system could provide rapid data acquisition and, with appropriate timing electronics, could allow for real-time lifetime imaging.
In cases of multispectral data, that is, samples with multiple stains, the separate wavelength data must be demultiplexed or unmixed. Signal unmixing, in either SPAD or CCD-based systems, is generally conducted in one of three ways: 1. The fluorescence signal can be measured for a selected discrete wavelength by use of a bandpass filter located in the fluorescence channel. Each measurement is recorded separately for each available bandpass filter to produce a set of images at each discrete wavelength [29,30]. The resultant images are then overlaid. This requires sufficient separation of the emission wavelengths that are being selectively filtered, with the biggest drawback being the requirement for multiple measurements on the same sample. Consecutive fluorescence measurements can lead to several practical issues: each image is susceptible to image registration errors due to moving parts (i.e., misalignment between successive exposures); increased prevalence of motion artefacts in live samples; and increased likelihood of sample bleaching of sensitive biological samples that are unable to withstand repeated exposures. 2. When fluorophores exhibit sufficiently distinct decay times, they may be differentiated by their time-of-arrival using fluorescence lifetime measurements [31,32]. In this case it is important for there to be sufficient difference in the fluorescence half-lives of each fluorophore such that distinct decay times may be determined. This approach is less effective when distinguishing fluorophores whose lifetimes are very similar. 3. Multispectral fluorescence can be split into its individual spectral components with the addition of either diffraction-based (eg grating) or refraction-based (eg prism) optical elements. Each spectral component may then be directed to a different detector, or different section of a detector array [10,[33][34][35]. This approach mitigates the drawback of filtering which by necessity can remove a large proportion of signal photons at each filter. By simply redirecting the light, the majority of photons are retained. This approach does require a high level of alignment accuracy of the additional optical components, and the additional elements may result in a more bulky system. Furthermore, the requirement in most cases of multiple separate detectors further increases the system size and cost. The different spectral paths may be directed onto different regions of one large-format detector array; however, this then reduces the available image resolution.
The use of colour filter arrays (CFAs) in conjunction with cameras is less common in fluorescence imaging for several reasons, one of which is the requirement for bespoke designs for particular wavelengths, pixel sizes and array formats. Existing thin-film or dye-based CFAs are fabricated using multiple lithographic stages for each unique spectral filter used, which typically limits filter arrays to a maximum of three filter types in mass manufacture processes. This fabrication complexity makes bespoke designs difficult to implement, when considering the range of possible bandpass filter combinations required, and varying pixel dimensions and array formats possible. Recently, however, high efficiency plasmonic metasurface spectral filters have been developed for multispectral imaging in the sparse photon regime [36,37]. Requiring only a single lithographic step, bespoke CFAs of this nature may be created without any major process modifications, with the possibility to include filters designed for any number of wavelengths, and for widely different pixel dimensions.

| System description
For the experiments presented in this paper, an Olympus Cell IX81 self-contained fluorescence imaging system was used as shown schematically in Figure 1. This microscope used a halogen lamp for full-field white-light illumination and a selection of continuous wave 100 mW laser diode sources were available for fluorescence excitation. A 150× magnification microscope oil-immersion objective (1.45 NA) was used. Back reflections from the laser illumination were attenuated in the detection channel by use of a quad-band filter with central transmission bands at λ=430, λ=530, λ=615, λ=675 nm. The full microscope component list is given in Table 1. The EMCCD that was normally used was replaced by a SPAD detector array described below.
The detector replacing the EMCCD is a 64 × 64 format silicon SPAD array with 61.5 μm pixel pitch and 11.6 μm side square active area [38], shown in Figure 2A. The relatively small active area of this device presents an obvious disadvantage in terms of overall detection efficiency; however, there are several approaches to increase low fill-factors in SPAD arrays. High concentration-factor microlens arrays are a simple and effective strategy [39,40], while advances in 3D stacking the in-pixel electronics [41,42] is a highly effective long-term solution.
Atop the SPAD sensor was a plasmonic metasurface CFA, integrated to the SPAD detector array using an ACCμRA100 flip-chip bonder. The metasurface was composed of a series of circular and elliptical nanoholes written using electron-beam lithography, and etched into a thin film of aluminium deposited on a glass substrate. Filters were designed to approximate red, green and blue wavelengths centred at λ=450, λ=540, λ=670 nm, and arranged in a random mosaic pattern as shown in Figure 2B. More details about the metasurface design and performance can be found in reference [37].

| Samples
Four biological samples were selected to demonstrate the use of the metasurface mosaic filter array in the acquisition of fluorescence images.

| Data acquisition
In each case, 50 frames were collected at 5 ms exposure per frame in photon-counting mode, with the illumination sources in each case applied simultaneously. The power levels of each laser were adjusted to provide similar levels of fluorescence in each channel. Background measurements were recorded and "hot pixels" with excessive dark counts were identified [45]. These hot pixels were discarded and these empty pixels were addressed during the image reconstruction procedure described below.

| Image reconstruction
In order to extract spectral information from a single intensity value per pixel, the recorded counts must be separated into their component contributions. We model the intensity recorded in the n th pixel, I(n), as a linear combination of L wavelengths (L = 3 in this case, for the RGB colours) plus the background level, B(n), according to Equation (1): where T i (n) is the transmission coefficient at pixel n and i th wavelength, and I i are the contributing intensities of each RGB component of incident light. These contributing intensities {I i (n)} i = 1,2,3 were recovered using an inpainting (i.e. matrix completion) algorithm, based on the fact that the coefficients {T i (n)} and average background levels {B(n)} are known (calculated during calibration). Despite Equation (1) involving more unknowns than equations (3 independent unknowns per pixel here), the algorithm is able to reconstruct multispectral images by considering the spatial correlation between neighbouring pixels (which present similar spectral features, except at object boundaries). The use of three different spectral filters over the array results in a maximum loss of information of 2/3rds over the frame, which can cause spatial blurring at object boundaries, and spectral uncertainty in the presence of very small features. These artefacts can be partly accounted for in the reconstruction algorithm using the regularisation used, but not completely avoided. More sophisticated regularisation strategies could further reduce these artefacts at the expense of a more computationally intensive algorithm. This maximum 2/3rds loss of information per channel is, however, reduced in this case through the use of a uniform, yet random, distribution of each filter set across the array, which has been shown to reduce these types of errors [46]. The spectral overlaps between filters and non-zero out-of-band transmission further reduces the losses. A more detailed description of the algorithm is provided in references [37], [47] The recovered intensity contributions from each wavelength may be considered to represent the three constituents of the RGB colour model [48] and combined accordingly to provide the "true" colour at each pixel. Unlike our previous work described in [37], an intensity calibration for each RGB component is not possible due to differing emission strengths of the fluorescences in each sample. Therefore the relative intensity weightings of each RGB component were manually adjusted in each case to provide the highest visual clarity. Figure 3 shows the colour image reconstruction from simultaneous multi-spectral fluorescence measurements of the four different samples with accompanying examples either from literature, or using a sCMOS camera. From Figure 3A a number of Leishmania parasites are clearly identifiable (four small dark features) along with the larger native macrophage stained a dark purple, while the surrounding cytoplasm retains a lighter shade. Figure 3B shows a close-up of the Convallaria root with its distinctive cell wall structure clearly visible with a strong red fluorescence. Figure 3C,D shows the fluorescence image reconstruction of BPAE cells. Given the low pixel format (64 × 64) and the large pixel size (61.5 μm 2 ) on this SPAD array, the fluorescence imaging was also performed with an additional 1.6× optical magnification applied, bringing the final image pixel size to 250 nm. This was done to provide higher spatial resolution in order to aid with identification of smaller multi-colour features within the sample which may not be distinguishable with the larger 400 nm image pixel size. However, it is worth noting that this increased magnification was achieved by introducing an additional optical element into the light path and therefore reducing the amount of signal reaching the detector. As SPAD detector array technology matures, pixel sizes are generally decreasing, while pixel array formats are increasing [49]. It is therefore likely that the use of a SPAD with smaller pixels could provide similar resolution without the need of additional optical elements, thereby producing significantly better results in terms of image quality and resolution. For sample C two illumination lasers were used simultaneously, red and blue for the unmagnified image ( Figure 3C(ii)); blue and green for the magnified image ( Figure 3C(iii)). Sample D was illuminated with all three (red, green, blue) lasers simultaneously. In each case, the cell nuclei are clearly visible, exhibiting blue fluorescence emission, with two nuclei in Figure 3D(ii). The green fluorescence shows tubulin filaments, while the red fluorescence represents mitochondria. In Figure 3C(iii) we can see a cell undergoing mitosis, while Figure 3D(iii) shows tubulin filaments in greater detail.

| RESULTS AND DISCUSSION
To compare the SPAD data with the sCMOS data, we plot the raw intensity of a similar area on the Convallaria. Different locations were used to prevent sample bleaching affecting the relative signal strengths. A total 1 ms exposure was used in each instance. Figure 4 shows two similar regions as detected by the SPAD, A, and the sCMOS camera B, and an intensity cross-section of a row of pixels intersecting the fluorescence on each. To plot roughly the same area over the same number of pixels required the sCMOS data to be downsampled by a factor of 7 (ie a 480 × 480 pixel region became 64 × 64 pixels). This was done by averaging each 7 × 7 block of pixels. Signal-to-noise (SNR) ratios are found to be 2 on the SPAD and 3 on the sCMOS, while the variation in the F I G U R E 3 Colour reconstruction of biological samples; A, Leishmania parasite in murine macrophage compared to p-i-n photodiode scanning system [50]; B, Convallaria cells alongside scientific CMOS (sCMOS) image; C, bovine pulmonary artery endothelial (BPAE) cells (slide #1) with blue and red illumination only (middle), and blue and green illumination only (right), alongside sCMOS image (left); D, BPAE cells (slide #2), alongside sCMOS image (left). Dashed lines represent approximate fields-ofview on the example images, which are provided for visual comparison raw data is much higher in the SPAD. A number of factors impact the relative quality observed in these data. The SNR on the SPAD is affected by a higher background leakage due to ambient light, as the SPAD was not fixed within the enclosed system unlike the sCMOS camera. The signal strength was also reduced by the presence of the additional 1.6× magnification lens used for this measurement. The large variance one can see in the intensity cross-section is caused by the presence of different filters over adjacent pixels, as this presentation shows the raw data before the algorithm extracts more accurate colour information.
To ensure the demonstrated colour reconstruction can be exploited in the sparse photon regime, the Convallaria data was examined under reduced exposure duration. Data were collected with exposure durations, t, reduced down to as low as 10 μs per frame. A sample of BPAE was also examined under various exposure conditions but with the excitation lasers reduced to below 10 mW of power, the approximate threshold for photodamage to occur in particularly sensitive samples [51][52][53]. We are able to count the average number of detected photons per pixel, ppp, necessary to reproduce high fidelity colour images from sensitive biological samples. Figure 5 shows reconstructions of the Convallaria using data representing total exposure times of t=250, 50 and 30 μs; and BPAE data at t=100, 10 and 5 ms-the longer exposure times necessitated by the lower excitation power. For both samples, the same data are presented for each exposure duration, but with more frames summed to provide the longer exposure times.
It can be seen that a reasonable image reconstruction quality is maintained down to as few as 2 − 3 photons per pixel over the Convallaria image, with total exposure durations of the order 10 μs. With reduced laser power, the BPAE sample shows reasonable multispectral image reconstruction with an average of 15 photons per pixel with the different regions easily identifiable. In these experiments, the average background and detector dark counts contribute 600 counts per pixel per second. In the sparse, photon-starved regime, these spurious counts may not be correctly accounted for by the background subtraction, and therefore can impact the restoration as they must still be allocated to the three RGB channels by the algorithm. Figure 5C, which provides a cross-section of the BPAE reconstruction by filter channel, shows this effect as a higher proportion of the signal is incorrectly attributed to the red channel in the sparse photon regime. In this case the 5 ms data represent a single frame, with the 100 ms data a summation of 20 frames. This "shot noise" can therefore be expected to cause significant image degradation in data of comparable signal level, setting a lower limit on the number of photons required to reconstruct a high quality image. This lower limit can be reduced by using a higher quality SPAD with a lower DCR, or cooling the SPAD array. The SPAD array in this case exhibits a relatively high dark current at room temperature (3.3 kHz mean); however, state-of-the-F I G U R E 4 Intensity comparisons between A, single-photon avalanche diode (SPAD) and B, scientific CMOS (sCMOS) data at similar locations under 1 ms exposure duration. C, Signal crosssection across the red (SPAD) and blue (sCMOS) lines indicated on A and B F I G U R E 5 Colour image reconstruction of Convallaria, A and bovine pulmonary artery endothelial (BPAE), B fluorescence under decreasing exposure duration, t. ppp represents the average photons per pixel, calculated only from those pixels contained within the regions of interest highlighted by white dashed lines in the leftmost image for each sample. The red lines show the pixel rows from which red, green and blue relative intensity cross-sections are taken for, C art SPAD arrays with Peltier cooling could conservatively reduce the detector's noise equivalent power by at least a factor of 50 [54]. Integrating the device fully into the microscope to reduce the ambient light entering the detector would also significantly improve sensitivity. A combination of these system improvements would allow for a reduction in both illumination power and exposure duration.

| CONCLUSION
We have demonstrated fluorescence imaging at multiple wavelengths using data acquired simultaneously using a single-photon avalanche diode detector array with integrated high-efficiency plasmonic mosaic colour filter array. This approach could have significant advantages in allowing rapid fluorescence measurements of different fluorophores. The particular SPAD array used did not have the necessary timing electronics to measure fluorescence lifetimes, however, this initial study shows that the extension to lifetime studies can be made using a similar spectral unmixing approach. The development of high-efficiency plasmonic metasurface-based spectral filters provides bespoke spectral unmixing capability in a single measurement, without the need for iterative sequences of measurements at numerous wavelengths using different optical filter combinations. We have demonstrated that sensitive biological samples may be probed with accurate reconstruction possible from as little as 30 μs of excitation laser exposure. This demonstrated sensitivity, combined with the frame-rate benefits of SPAD arrays over CCD-based technologies, allows for a significantly shorter measurement time and could facilitate the creation of high-speed movies of dynamic processes.
With reduced intensity multispectral illumination it was possible to reconstruct high quality, full colour images from only 10 photons/pixel in the order of a few ms. This relates to approximately three orders of magnitude faster than automated sequential three-colour illumination systems, which typically require a few seconds to switch between filters. Furthermore, this is achieved without incurring the size and cost penalties related to splitting the signal into its components, such as additional bulky optics and the plurality of detectors generally required for this modality. Further development of these detectors in conjunction with high-efficiency plasmonic spectral filtering will have significant implications for live imaging of fast biological processes, which are beyond the temporal resolution of traditional systems. Future work will take advantage of existing SPAD technology by considering SPAD detector arrays with picosecond timing resolution to enable simultaneous fluorescence lifetime measurements at multiple wavelengths. In addition, the use of larger format SPAD detector arrays with reduced pixel size will provide for higher resolution imaging.

ACKNOWLEDGMENTS
The imaging experiments were conducted using the facilities provided by the Edinburgh Super Resolution Imaging Consortium (ESRIC) at Heriot-Watt University. This work was supported in part by the Royal Academy of Engineering under the Research Fellowship scheme RF201617/16/31, and UK Engineering and Physical Sciences Research Council projects EP/N003446/1, EP/T00097X/1 and EP/S026428/1.

CONFLICT OF INTEREST
The authors declare no potential conflict of interests.