A Hybrid Solar Irradiance Nowcasting Approach: Combining All Sky Imager Systems and Persistence Irradiance Models for Increased Accuracy

The share of distributed solar power generation is continuously growing. This increase, combined with the intermittent nature of the solar resource creates new challenges for all relevant stakeholders, from generation to transmission and demand. Insufficient consideration of intra‐minute and intra‐hour variabilities might lead to grid instabilities. Therefore, the relevance of nowcasts (shortest‐term forecasts) is steadily increasing. Nowcasts are suitable for fine‐grained control applications to operate solar power plants in a grid‐friendly way and to secure stable operations of electrical grids. In space and time, highly resolved nowcasts can be obtained by all sky imager (ASI) systems. ASI systems create hemispherical sky images. The associated software analyzes the sky conditions and derives solar irradiance nowcasts. Accuracy is the decisive factor for the effective use of nowcasts. Therefore, the goal of this work is to increase the nowcast accuracy by combining ASI nowcasts and persistence nowcasts, which persist with the prevailing irradiance conditions, while maintaining the spatial coverage and resolution obtained by the ASI system. This hybrid approach combines the strengths while reducing the respective weaknesses of both approaches. Results of a validation show reductions of the root mean square deviation of up to 12% due to the hybrid approach.

DOI: 10.1002/solr.202100442 The share of distributed solar power generation is continuously growing. This increase, combined with the intermittent nature of the solar resource creates new challenges for all relevant stakeholders, from generation to transmission and demand. Insufficient consideration of intra-minute and intra-hour variabilities might lead to grid instabilities. Therefore, the relevance of nowcasts (shortestterm forecasts) is steadily increasing. Nowcasts are suitable for fine-grained control applications to operate solar power plants in a grid-friendly way and to secure stable operations of electrical grids. In space and time, highly resolved nowcasts can be obtained by all sky imager (ASI) systems. ASI systems create hemispherical sky images. The associated software analyzes the sky conditions and derives solar irradiance nowcasts. Accuracy is the decisive factor for the effective use of nowcasts. Therefore, the goal of this work is to increase the nowcast accuracy by combining ASI nowcasts and persistence nowcasts, which persist with the prevailing irradiance conditions, while maintaining the spatial coverage and resolution obtained by the ASI system. This hybrid approach combines the strengths while reducing the respective weaknesses of both approaches. Results of a validation show reductions of the root mean square deviation of up to 12% due to the hybrid approach.
increasingly important for energy grid control applications such as demand balancing/managing, power scheduling, and unit commitment. [9] Proper demand management could reduce cost mainly by the reduction of otherwise required peaker plants. [7] Forecasts looking even further ahead are useful for scheduling of maintenance work. [9] In general, there are two main strategies to produce nowcasts/ forecasts. First, data-driven statistical approaches and second, physical approaches based on ground-based sky images, satellite images, or numerical weather predictions (NWPs). A wide variety of statistical approaches exist, covering the range from highly resolved local nowcasting approaches to coarse forecasts looking multiple weeks ahead. [10] Ground-based sky images are suitable for local nowcasts. [11] Satellite-based approaches provide forecasts with a reasonable accuracy up to 6 h ahead, covering large continental areas with typical temporal and spatial resolutions of 15 min and 2-10 km, respectively. [12] More advanced satellite systems such as the Himiwari-8 or GOES-R reach spatial resolutions of 0.5 km 2 and temporal resolutions down to 5 min. [13] NWPs are based on multiple differential equations describing relevant physical and chemical processes from an initial state, typically suitable for forecasts with temporal and spatial resolutions of ≥1 h and ≥1 km, respectively, and horizons of multiple days ahead. [14] Forecasts are never perfect and always subject to uncertainties. Uncertainties may reduce the potential benefit of forecasts and lead to new challenges such as increasing solar curtailment and increased demand for balancing power. [10] Further potential effects include unfavorable assessment of electricity prices and increased maintenance cost due to accelerated degradation of technical facilities. Therefore, the reduction of forecast uncertainties is of great importance to overcome the challenges for substantial PV integration.
The aim of this work is to reduce the uncertainties of solar irradiance nowcast based on sky images, which are suited to represent intra-minute and intra-hour variabilities caused by clouds. For this task, sufficiently resolved sky images can be obtained by the socalled all sky imager (ASI) systems, consisting of upward-facing cameras with fisheye lenses. In recent years, methods have been developed to automatically detect clouds, [15] evaluate the cloud radiative effect, [16] and identify cloud motion vectors from such sky images. [17] If multiple cameras with overlapping viewing angles are available, cloud geolocalization can be performed using stereo photography. [18] A combination of such processing steps can be applied to derive irradiance nowcast. [19] Further methods of obtaining irradiance nowcasts are persistence approaches, which also serve regularly as control reference models. As the name implies, these approaches are based on maintaining the current state. The most basic global horizontal irradiance (GHI) persistence predictions are described by GHI p (t þ Δt) ¼ GHI(t). More complex persistence approaches may take future sun positions or clear sky indices into account. [20] The performance of persistence approaches is often competitive in intra-hour nowcasting, [20] considering the challenges of ASIbased nowcasting systems. [21] This work presents a real-timecapable hybrid ASI and persistence nowcasting approach, which combines the strengths while reducing the respective weaknesses of both approaches.
This article is structured as follows: Section 2 describes the utilized persistence, ASI, and hybrid-based nowcasting approaches. Section 3 introduces the used validation data set and the validation procedure. Results of the validation procedure are presented and discussed in Section 4. Finally, Section 5 concludes the findings.

Experimental set up and method
The persistence and ASI-based nowcasting approach as well as the hybridized nowcasting approach are described in this section. The given description of the ASI-based approach is limited to an overview, as it is by itself not the main subject of this work. A more detailed description can be found in the corresponding cited publications.

Persistence Nowcast
The utilized persistence nowcast is based on the work by Ineichen and Perez. [22] The prevailing Linke turbidity factor (TL) is calculated using the GHI typically measured by a pyranometer, h the height of the site, I 0 the solar constant, r 0 the average distance from the Sun to the Earth, r the current distance from the Sun to the Earth, α the solar altitude angle, and AM the air mass.
For the persistence nowcast, the TL is kept constant while α and AM are calculated for lead times >0 min. Subsequently, the GHI persistence nowcast is calculated with the corresponding rearranged Equation (1) and the future solar altitude angles.

ASI-Based Nowcast
A nowcasting system based on two off-the-shelf Mobotix Q25 surveillance cameras has been developed. The camera setup of the used nowcasting system is listed in Table 1. The ASI-based nowcasting system also uses ground-based measurements of direct normal irradiance (DNI) and diffuse horizontal irradiance (DHI), taken directly next to one of the cameras.
The final output of the ASI-based nowcasting system consists of GHI maps, covering an area of >60 km 2 with a constant spatial resolution of 20 m and lead times up to 20 min ahead. Figure 1 shows such a GHI map and the corresponding cloud model as well as all the needed processing steps. These steps include cloud detection with a convolutional neural network, [23] cloud geolocating based on a stereoscopic approach, [24]  cloud tracking with multiple sequential images and cross correlation, [24] shadow projection of current and predicted cloud positions via ray tracing, [25] and the analysis of the radiative effect by means of a probabilistic approach utilizing recent and historic cloud height and DNI as well as DHI measurements. [26]

Hybrid Nowcast
The ASI-based nowcasting system introduced in Section 2.2 creates irradiance maps. These maps incorporate the locations of cloud shadows predicted from the current observations by the cameras and the used irradiance measurement. Such a map at the time of the image acquisition is shown in Figure 2 (left site, "observation map"). The green cross marks the position of the used irradiance measurement. For this position of the observation map the irradiance will always match directly the known irradiance from the ground-based irradiance measurements. The remaining sections of the map receive irradiance information derived from the ASI images. Starting from the observation map, two distinct sets of nowcasts are created.
The first set of nowcasts uses the persistence approach. Thereby, the persistence nowcasting process from Section 2.1 is applied to each pixel of the observation map using the local estimation of GHI as input, creating 20 distinct persistence  www.advancedsciencenews.com www.solar-rrl.com nowcasting maps with lead times (LTs) from 1 to 20 min ahead (see Figure 2 top right). For the marked pixel this corresponds to the persistence of the ground-based irradiance measurements. The second set of nowcasted GHI maps is derived by the ASI-based approach as described in Section 2.2 (see Figure 2, bottom right). A sliding validation of recent historical nowcasts is performed for both nowcasting approaches using the observation maps of the last 5 min. For this purpose, the sliding root mean square deviation (RMSD_Sl) is calculated in real time according to Equation (2) for each pixel individually using the observation GHI maps ðGHI obs,px Þ of the past 5 min as reference for the corresponding GHI nowcasts GHI LT,px,j for each delivered lead time.
The index LT specifies the lead time, px the pixel ID, and the index j indicates the two distinct nowcasting approaches. www.advancedsciencenews.com www.solar-rrl.com t i describes the corresponding time stamp and n the selected time interval of 5 min in the ASI system resolution of 30 s. This validation approach represents an autovalidation between ASI observations and the corresponding nowcasting approaches, except for the pixel corresponding to the position of the used irradiance measurement station. A corresponding illustration of this autovalidation procedure is shown in Figure 3. Autovalidations should be treated with precaution, since intrinsic errors of the ASI system are not considered. In the present application, it is less critical as the autovalidation is only used to combine the different nowcasting approaches over the space covered by the GHI maps. The final overall validation of the distinct nowcast approaches, as described in Section 3.2, does not involve an autovalidation. Subsequently, current nowcasts GHI LT,px,j of both approaches for a given lead time are combined to the hybrid nowcast using the accuracy weighting approach described by Meyer et al. [27] according to Equation (3) for each pixel of the maps individually. An illustration of this approach is shown in Figure 4.

Validation Data Set and Procedure
This section includes a description of the used validation period as well as the validation procedure.

Validation Data Set
The ASI, persistence, as well as the hybrid nowcasting approach are validated at CIEMAT's Plataforma Solar de Almería (PSA) with eight reference ground-based stations distributed over an area of roughly 1 km 2 . The eight stations use ISO 9060 class A spectrally flat pyranometers and pyrheliometers and automated solar trackers with sun sensors. Reference values are used as 1 min averages, and the time stamps describe the end of the period. Figure 5 shows an aerial image of CIEMAT's PSA with blue markers for the eight reference positions and green markers for the used ASI positions. The irradiance measurement from the ground-based station next to the southwestern ASI serves also as input for the nowcasting systems.
The validation data set contains 62 days between 01.09.2019 and 30.11.2019. Figure 6 shows the GHI conditions within the validation data set as arithmetic average of the eight reference pyranometers. The validation period included a wide variety of   conditions from clear sky to scattered clouded with distinct ranges of irradiance fluctuations up to overcast. For the validation only time stamps with a Sun elevation angle ≥20 are used.

Validation Procedure
For the further benchmark, a purely persistence-based nowcast is derived from the ground-based station next to the southwestern ASI (see Figure 5) according to the procedure described in Section 2.1. This is the same station that provides the needed irradiance measurements to the ASI as well as the hybrid-based nowcasting approach. Thus, all nowcasting approaches benchmarked against each other had access to exactly the same irradiance measurements. The original ASI-based nowcast and hybrid nowcast are available in a 30 s resolution. 1 min temporal averages are created, concordant with the reference data and the purely persistence-based nowcasts.
The RMSD is used as an error metric for the benchmark between the different nowcast approaches. For each of the eight reference stations, the nowcast RMSD is calculated according to Equation (4) over the entire validation period (number of evaluated timestaps, m) discretized over the distinct lead times.
For this purpose, the nowcasted GHI is obtained from those pixels of the GHI maps that represented the positions of the eight reference stations (index st), in the case of the ASI-and hybridbased approaches. In case of the purely persistence-based nowcasting approach, no spatial information is available. Therefore, the same nowcast derived from the southwestern station is validated with all eight reference stations. Consequently, this gave the opportunity to evaluate the spatial suitability of a persistence nowcast derived from a single local measuring station, by examining the scatter of the RMSD across the different reference stations.
Furthermore, spatial aggregation effects are evaluated. Spatial averages are created from the GHI maps of the ASI-and hybridbased approaches, utilizing all pixels within the 1 km 2 area marked in Figure 5 by the yellow square that contains all eight reference stations. As the reference data for this spatial average, an arithmetic average of the eight reference stations is created. Again, no spatial information is available for the purely persistence-based nowcast approach. Nonetheless, the persistence nowcast is also validated against the arithmetic average of the eight reference stations.
The validation procedure described so far shows the overall performance of the nowcasting approaches. Such overall validation results give a first impression of the relative performance of the three approaches for the same data set. However, these overall results are inadequate for comparing validation results of distinct sites or even distinct data sets because the performance of the different nowcasting approaches depends strongly on the composition of the prevailing conditions within the data sets. Comparatively high accuracies are reached during steady conditions with low variability consisting of clear sky or overcast conditions. In contrast, lower nowcasting accuracies are reached during highly variable conditions, especially under conditions with fast-moving low layer clouds or under complex multilayer conditions. Overall, validation results for a given time interval without consideration of the prevailing conditions can lead to false impressions of the accuracy. Therefore, the validation data are discretized into eight distinct variability classes from clear sky (class 1) to overcast (class 8). This classification procedure is based on the work described in Schroedter-Homscheidt et al. [28] with the adaptations described in Nouri et al. [29] The classification is fully automatized and analyzes 13 distinct variability indices. This classification procedure assigns a class to each time  www.advancedsciencenews.com www.solar-rrl.com stamp based on the irradiance conditions of the previous 15 min. Table 2 lists a short description of the variability classes. Figure 7 shows the distribution of the variability classes as well as the average GHI for each class within the validation data set. Around 57% of the data comprise the low-variability conditions classes 1, 2, and 8. Intermediate and highly variable conditions comprise 34% and 9% of the data set, respectively. Most clear sky conditions at low solar altitude angles are classified in class 2, which results in comparably low GHI values within this class. The average GHI over the entire data set is 384 W m À2 .

Results and Discussion
The overall validation results without and with spatial aggregation including all days and variability classes are shown in Figure 8. As expected, the deviations increase with rising LTs for all the approaches. The RMSD curve flattens notably with increasing LT. Among the individual reference stations, the persistence approach shows a strong scatter in the low LTs of 1-4 min. This is especially pronounced for LT 1 min, with a spread of 23 W m À2 in between the reference stations. The lowest deviations belong to the southwestern reference station next to one of the ASIs. This is to be expected, as the persistence nowcast is derived from the measurements of that station in particular. The persistence approach underperforms for these low LTs already at distances beyond 500 m from the origin of the persistence nowcast. For the higher LTs (>4 min), this effect caused by the spatial distance between the stations is no longer visible. Variations between the stations quickly converge to a spread of 3-5 W m À2 . The ASI and hybrid approaches show a similar spread of 3-5 W m À2 over all LTs. When comparing the persistence-with the ASI-based approach, it is apparent that up to a lead time of 5 min the ASI-based approach performs similarly to the lower bandwidth of the persistence approach. The exception is the LT of 1 min, where the persistence nowcast achieves overall the lowest deviation for the southwestern reference station (origin of the persistence nowcast). Beyond LT of 5 min and below LT of 16 min, the ASI-based nowcast shows a clear advantage, with deviations up to 6% lower than those of the persistence-based nowcast. Both approaches perform similarly for LTs beyond 16 min. In contrast, the hybrid nowcasting approach shows a significant performance improvement. An average reduction of around 11.7 and 9.2% in RMSD is visible for all LTs compared to the persistence-or ASI-based approach, respectively. When taking the spatial aggregation effects into account, the hybrid approach remains clearly the most performant approach, outperforming the persistence-and ASI-based approach on average by 13.5% and 10.8%, respectively. Furthermore, the spatial aggregation has a significant impact on the deviations compared to the deviation based on individual reference stations. Overall reductions in RMSD are visible by 7.7%, 8.0% and 9.6% for the persistence, ASI, and hybrid approaches, respectively. Figure 9-12 show the validation results discretized over the variability classes. Unsurprisingly, the lowest RMSD and MAD values are observed for the nearly clear sky conditions (class 1 and 2). Furthermore, as expected, the persistence approach shows especially low error metrics for these least variable conditions. However, two anomalies are visible for the ASI Figure 8. Overall RMSD over lead times of all three approaches for individual reference stations as well as spatial averages.
www.advancedsciencenews.com www.solar-rrl.com system at variability class 1. These anomalies are comparatively pronounced for the RMSD and to a lesser degree also visible for the MAD. The origin of these partially amplified deviations are most likely outliers caused by false positive detections of high Figure 9. RMSD discretized over variability classes 1 to 4 (see Table 2) and LTs of all three approaches for individual reference stations as well as spatial average. Figure 10. RMSD discretized over variability classes 5 to 8 (see Table 2) and LTs of all three approaches for individual reference stations as well as spatial average. Figure 11. MAD discretized over variability classes 1 to 4 (see Table 2) and LTs of all three approaches for individual reference stations as well as spatial average. Figure 12. MAD discretized over variability classes 5 to 8 (see Table 2) and LTs of all three approaches for individual reference stations as well as spatial average.
www.advancedsciencenews.com www.solar-rrl.com layer clouds. Nevertheless, also for the ASI approach the error metrics remain comparatively low under variability class 1 conditions. The third low-variability condition (class 8, overcast) also shows low RMSDs and MADs. However, especially for the high LTs, the error metrics are by a factor of up to three higher than for class 1 conditions. All three nowcasting approaches show similar error metrics for class 8. For the intermediate-variability condition, class 3, the ASI-based approach shows a significant advantage compared to the persistence approach in terms of RMSD. In terms of MAD, the ASI advantage is rather small. The remaining intermediate variability classes show a similar performance in RMSD between the ASI-and persistence-based nowcasts, with a slight advantage for the persistence in class 5 and a slight advantage for the ASI in class 7. In terms of MAD, the persistence approach shows a more notable advantage compared to the ASI approach. Both approaches are outperformed by the hybrid approach in all intermediate variability conditions. As expected, the persistence approach yields high RMSDs and MADs for the highly variable conditions classes 4 and 6. The ASI approach outperforms the persistence approach clearly in these cases, especially for class 4, whereas the ASI and hybrid approaches show similar performances. The significant overall advantage of the hybrid approach is due to the constant adjustment of the weighing factors based on the sliding validation, as described in Section 2.3. This weighting process, favors the persistence approach under low and intermediate variability conditions, where false positive or false negative cloud detections of the ASI approach could lead to higher deviations. In contrast, the persistence approach by definition is not capable of modeling any variabilities except due to the changing Sun position in the sky. Therefore, the ASI approach will be favored most often under variable cases with a scattered cloud cover.
Since the hybrid approach outperforms almost under all conditions the other approaches, it does not appear beneficial to select a method other than the hybrid approach as a function of the prevailing conditions. Figure 13 shows the average spread over all lead times in RMSD between the eight reference stations as well as the average impact of spatial aggregation on the RMSD. The lowest spread in absolute RMSD between the eight distinct reference stations is visible for the overcast conditions (classes 7 and 8) with values between 4 and 6 W m À2 . This is due to the mainly homogeneous cloud cover leading to a low to intermediate variability combined with a low GHI. Under such overcast conditions, uncertainties of the processing steps, cloud geolocation and tracking have hardly any influence on the nowcast uncertainties. In comparison, the clear sky and low-variability conditions classes 1 and 2 show a more pronounced spread in absolute RMSD with values between 6 and 9 W m À2 and 9 and 15 W m À2 for class 1 and class 2, respectively. However, in relative terms, class 1 has clearly the lowest spread in RMSD between the reference stations due to the overall high GHI. The most significant spread between the reference stations is visible for classes 3 and 4. This is to be expected as these conditions are related to the highest GHI values after class 1 combined with an intermediate to high variability.
Overall the spatial aggregation effects have a benefit over all classes. This benefit is rather low for the classes with lowvariability conditions, especially for class 8. The strongest impact is visible for the intermediate and highly variable conditions classes 3, 4 and 6, which are also the classes with the highest overall deviations. The importance of spatial effects becomes clear when considering the size of utility-scale PV power plants. According to Ong et al., [30] the average total land use per megawatt in the United States is around 0.03 km 2 , whereas the average capacity per power plant is around 9 MW. [31] On the upper end of the scale, solar parks exist consisting of multiple power plants with total capacities >2 GW, covering areas >50 km 2 . [32]

Conclusion
An ASI and a purely persistence-based nowcasting approach were benchmarked. In addition, a hybrid nowcasting approach was developed and added to the benchmark. The hybrid approach combines the spatial nowcast from the ASI approach with a spatial persistence nowcast derived from the ASI observations.
The validation procedure is conducted with eight distinct reference stations distributed over an area of roughly 1 km 2 . The persistence-based nowcast is derived from one of the eight reference stations, which is located in the southwestern corner of the validation space. If only the measurement station used for the creation of the persistence forecast is applied for the validation, the purely persistence-based approach shows a performance similar to that of the ASI-based approach for LTs below 6 min. Beyond this point, a clear advantage of the ASI-based approach is visible. However, within the first 4 min ahead, the persistence www.advancedsciencenews.com www.solar-rrl.com nowcast accuracy varies strongly between the reference stations. The performance of the persistence approach decreases strongly with increasing distance from the origin of the persistence nowcast, with an increase in RMSD of up to 23 W m À2 for reference stations at a distance beyond 500 m. For the higher LTs (>4 min), this effect is less significant. Both the purely persistence-and the ASI-based approaches are clearly outperformed by the hybrid approach over all LTs. The performance of the nowcasting approaches also depends strongly on the conditions at hand. Between the ASI-based and purely persistence-based approaches, a clear difference is visible when discretizing the validation data in the eight classes with distinct irradiance variability conditions. As expected, the persistence approach shows the best performance for low variability conditions, whereas it is outperformed by the ASIs during highly variable conditions. This is based on the fact that the persistence approach by definition cannot model any variability except due to the changing Sun position in the sky. The clear division of strengths within distinct conditions is exploited by the hybrid approach, which clearly outperforms the ASI and persistence approaches under almost all conditions. Overall, RMSD reductions between 9% and 12% are reached by the hybrid approach. The only exceptions are highly variable conditions with an intermediate clear sky index represented by the irradiance variability class 4. Under these conditions, the ASI-based approach clearly outperforms the purely persistence-based approach, whereas the hybrid approach achieves results similar to the ASI-based approach.
Furthermore, for all three approaches the influence of spatial aggregation on an area of 1 km 2 was analyzed. There was a significant beneficial impact on the overall performance of all the approaches, with overall reductions in RMSD by 7.7%, 8.0%, and 9.6% for the purely persistence-, the ASI-, and the hybrid-based approaches, respectively.
The presented real-time capable hybrid combination method is a promising approach to improve ASI-based irradiance nowcasts. Further particularly promising potential for improvement in ASI-based nowcasting lies in the geolocation and tracking of clouds, especially under complex multilayer cloud cover conditions. Dense networks of ASIs with overlapping viewing angles can lead to further notable performance enhancements of ASIbased nowcasts. [33] Such networks can also increase significantly the nowcasts' horizon while maintaining the same resolution. Future research will evaluate in detail the nowcasts potential to improve the performance of electrical grids and power plants under variable conditions, despite the nowcasts uncertainties.