A Concise Overview on Solar Resource Assessment and Forecasting

The field of solar energy can be broadly categorized into four parts: (1) solar resource assessment and forecasting, (2) photovoltaic (PV) technology, (3) concentrating solar power (CSP) technology, and (4) solar heating and cooling. Insofar as solar radiation is concerned, solar engineering takes a vital role in connecting atmospheric science, which deals with the atmospheric chemistry and physics governing the amount of solar radiation reaching the Earth’s surface, and downstream applications, such as grid integration, daylighting, or heating, ventilation, and air conditioning (HVAC). However, unlike atmospheric science, electrical engineering, architecture, or mechanical engineering, it is not entirely clear whether or not solar engineering can be viewed as a subject on its own—very few universities offer a curriculum on that, and very few people would attain a certificate that says "Bachelor of Solar Engineering" at the end of the day. That said, as long as climate change and carbon neutrality are of interest, it is a fact that solar engineering would be involved in one way or another. On this point, one must be aware of the interdisciplinary nature when making scientific inquiries regarding the use of solar energy.

Take grid integration of solar energy, for instance, power system operators require information on future solar generation and electric load at different time scales and horizons, in order to perform unit commitment—an integer programming problem determining which thermal generator should be turned on at which instance. Clearly, the sciences and technologies involved in this task are multifaceted. First and foremost, there is atmospheric science, which is required to address the questions of when, where, and how much solar radiation is available. Next, solar engineering must be involved, since it deals with the conversion from solar radiation to solar power. Last but not least, power system engineering knowledge is needed in order to balance the (conventional thermal plus renewable) generation and (electric, heating, and cooling) load in an operational fashion. On top of all those, modern scientific techniques, such as statistics, machine learning, or mathematical optimization, almost always play a part.

Through the above elaboration, one can readily understand the challenge here: no one would be able to comprehensively acquire all knowledge pertaining to grid integration. And the same can be said for daylighting, HVAC, or any other application of an interdisciplinary nature. The undercurrent of resistance, as in many problems of a like form, is chiefly due to our limited time—so much has been collectively known by humankind, that a single lifetime is not sufficient to reach all frontiers of knowledge. Atmospheric science, as traditionally conceived, is the study of the Earth's atmosphere. Customarily, once theories in regard to atmospheric chemistry and physics are developed and understood, the job of atmospheric scientists is considered done. On the other hand, solar engineers are concerned with applications, hardware, and execution. During that process, little attention has been paid to the suitability and validity of scientific theories on which their engineering is based. There really is not much surprise to the reasons that have led to the status quo—people in various communities went to different schools, received training under different thinking styles, and are employed by departments of different agendas. The sheer amount of effort that would be required to convert to convert from one profession to another is monumental.

We all know it is generally beneficent to know more, but consider this question: how often do we spend time on reading relevant but secondary literature as to our own research domain? Priority is an interesting notion, in that, it is supposed to be solely internal to individuals. It is human nature to prioritize those tasks that are familiar to us and, in parallel, postpone those less familiar ones. To that end, if we are to deviate from this rule of thumb, others who hold fast to it may consequently produce more results and thus attain more recognition. This is thought to be the inquiry of interest: should we invest time and energy in things we are partially interested in but are not good at? Involution, a popular word of the Chinese urban dictionary, which refers to the situation where peer pressure forces one to take (or, not to take) certain actions that may or may not lead to a favorable outcome, is sufficient in narrating the scenario.

One remedy to counter the effect of involution is to reduce the amount of effort that is required to achieve certain goals. More formally, in line with the theory of economics, the trick is to lower the opportunity cost. One possible action of this kind is to summarize the best practices in one field to researchers and professionals in another field, e.g., through review papers. However, reading review papers is still an inefficient approach, since there are all too many of them—there are hundreds, if not thousands, of review papers published on solar resource assessment and forecasting alone—especially under today's "publish or perish" regime, which has been commonplace in most academic domains. In this regard, it is thought that überreview has now become absolutely necessary. One may interpret an überreview as a review of reviews. It is on this account that an überreview on solar resource assessment and forecasting is herein presented.

Presenting an überreview is by no means a simple task, because the literature contains many outdated, duplicated, and non-representative reviews that can be misleading or even harmful, particularly to those who are not familiar with the domain. In fact, even highly cited recent textbooks and handbooks can be outdated, and may contain questionable information. Since solar resource assessment and forecasting is a fast-advancing field with many parallel works, this überreview is composed of carefully selected and most representative reviews from credible sources that could fully reflect the state-of-the-art.

The idea central to solar resource assessment resides in identifying suitable and reliable data, without which no conclusion made can be deemed valid. Data for solar resource assessment present in three main forms: (1) ground-based measurements, (2) remote-sensing retrievals, and (3) output of numerical weather prediction (NWP) models. Among these three forms, carefully calibrated ground-based measurements are most accurate, followed by remote-sensing retrievals, whereas NWP output is the least accurate.

Living in an age of data explosion, one must not forget that solar radiation data was once scarce. Before the advent of modern remote sensing and NWP, researchers used to rely on low-accuracy empirical models for the estimation of solar radiation, such as the Ångström–Prescott type of models, which are based on sunshine duration. As compared to the current data practice and ways of estimating or retrieving solar radiation, sunshine duration data are of poor and inconsistent quality, and empirical correlations obtained at one location rarely apply to another. Therefore, many researchers, such as Gueymard et al. (2009), had advocated the termination of use of such models, years ago. As of now, satellite-to-irradiance algorithms and improved parameterization of NWP models have long become mainstream, albeit numerous research issues remain. In this section, data and modeling issues related to solar resource assessment are reviewed in sections 2.1 and 2.2, respectively.

Ground-based measurements are made using radiometers, and the respective science is known as radiometry. Whenever radiometry is of interest, the textbook by Vignola et al. (2019) always presents itself as a useful reference. In terms of instrumentation, that is, pyranometer and pyrheliometer, different radiometers are subject to different measurement uncertainties and performance. Due to the high cost of research-grade radiometers, it is exceptionally luxurious to collocate more radiometers than needed at a single site. In that, the review and intercomparison of 51 collocated radiometers, as presented by Habte et al. (2016), is one of a kind. Ground-based data can be used to validate the other two forms of data. However, its own quality control (QC) must be first conducted, to ensure that the baseline for any subsequent validation is legitimate. Unfortunately, there is not any QC routine that can be deemed universal, but the basic one set forth by Long and Shi (2008) has gained most acceptance and is used by the Atmospheric Radiation Measurement (ARM) and the Baseline Surface Radiation Network (BSRN). Figure 1 shows a diagnostic visualization of potential data problems in ground-based measurements. Detailed interpretation of this plot is not within the scope of this review, nevertheless, one can see the level of complexity that is typically involved during QC of irradiance data—it is by no means just applying a few statistical filters, like what most people do, instead, much domain knowledge is required to justify the validity of the data under scrutiny.

Figure 1.  Some visualization to facilitate quality control of irradiance data. See Forstinger et al. (2021) for detailed interpretation of these plots.

Solar engineers are interested in four types of radiometry measurements: (1) global horizontal irradiance (GHI); (2) beam normal irradiance (BNI)^①; (3) diffuse horizontal irradiance (DHI); and (4) global tilted irradiance (GTI). Whereas the energy production of PV systems depends on GTI, that of CSP depends solely on BNI. It is well known that GHI can be split into BNI (after the modification of cosine of zenith angle) and DHI. Nonetheless, since the definition of what constitutes the beam component is a human convention, some issues regarding the circumsolar region—the vicinity of the Sun disk—exist, and a good summary of those issues can be found in Blanc et al. (2014). In fact, measuring the beam radiation component can be considered as a study domain on its own, namely, directional radiometry [see (Mishchenko, 2011), for review], which is an active research field in the area of atmospheric measurement technique, because lots of information about aerosols and clouds can be derived from circumsolar measurements. As for GTI, it is not only affected by horizontal irradiance components, albedo also plays a vital part; for albedo-related topics, the reader is referred to Gueymard et al. (2019). In any case, publicly available, research-grade, ground-based radiometry data are rare as compared to data of other meteorological variables such as temperature, humidity, or precipitation. For instance, there are about 100 radiometry stations in China, however, measurements of the aforementioned basic meteorological variables are available at more than 50 000 stations. On this point, Chapter 6 of Sengupta et al. (2021) contains a rather complete list of ground-based radiometry data sources.

In contrast to ground-based data, satellite-derived irradiance covers all locations on Earth that are between $ \pm60^\circ $ latitudes, see Fig. 2. For higher-latitude locations, satellite-derived irradiance is also available, but only comes at a lower temporal resolution. This is because satellite-derived irradiance for mid and low latitudes is retrieved from remote-sensing images taken by geostationary satellites, whereas for high-latitude irradiance estimations, they come from data of polar orbiters. Given the fields-of-view of different weather satellites, as well as the ownership of these satellites, satellite-derived irradiance products are developed by different agencies and weather centers, and thus are heterogeneous. Not only are the retrieval algorithms powering the products different, accuracy also varies greatly across products, and across locations, time periods, and sky conditions. This is precisely why a somewhat major effort has been paid to validating these satellite-derived irradiance products, among which the work by Yang and Bright (2020) can easily be regarded as most comprehensive, at the time of writing.

Figure 2.  Five geostationary weather satellites jointly cover all locations on Earth between ±60° latitudes. Gridded irradiance estimates can be derived from the visible- and infrared-channel images captured by these satellites. (Other meteorological satellite series, such as Fengyun, are not shown, since their field-of-views overlap with the ones in the figure.) Data source: National Solar Radiation Data Base.

Satellite-to-irradiance algorithms can be broadly categorized into those that rely on radiative transfer and those that do not. Huang et al. (2019) provided a comprehensive review elaborating the pros and cons of the two categories of method. And a more detailed description regarding the treatment of clouds during the retrieval can be found in Miller et al. (2018). Notwithstanding, regardless of which algorithm is used, the satellite-derived irradiance products are always limited by the native resolutions of the original satellite imagery. Whereas previous-generation satellites have a 60-min–10-km native resolution, the native resolution of the latest-generation satellites has reduced to 10 min 1 km, offering more opportunities for operational applications of satellite data in solar engineering.

The output of NWP models is of two kinds, one forecast and the other reanalysis. Operational NWP models issue forecasts a few times a day, over forecast horizons of a few days, at a regional or global scale. From a statistics point of view, weather forecasts are multivariate time series of lattice processes, whose spatio-temporal trajectory is governed by the physical laws of the atmosphere. On this note, the textbook written by Cressie and Wikle (2015) is inspirational, in terms of linking statistics and physical science. Since atmospheric scientists are familiar with this topic, only a few notable, general, and educational reviews are suggested here: Bauer et al. (2015); Müller et al. (2021); McNeal et al. (2021). What is more important, instead, is how to parameterize NWP models such that the quality of its radiation output can be enhanced; the reader is referred to Jimenez et al. (2016) for a list of issues and challenges.

NWP produces forecasts, but the operational models are constantly undergoing changes and developments. In contrast, reanalyses use “frozen” models and produce estimates of weather variables over a period typically spanning a few decades. The most well-known global reanalyses include the ECMWF Reanalysis, Version 5 (ERA5; Hersbach et al., 2020) and Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA-2; Gelaro et al., 2017), which have been used in countless ways by researchers from various fields. With no exceptions, these reanalyses have been shown to be useful in solar engineering as well, particularly when measurements of atmospheric variables, such as aerosol optical depth or surface albedo, are required but are unavailable. At the moment, the literature is short of a review on how reanalysis data are involved in solar engineering.

Many calculations in the field of solar engineering begin with the solar constant. The solar constant is obtained through calculating the average of total solar irradiance (TSI), which is the Sun's instantaneous output, over a long period of time, typically over a few decades. Whereas the overall concept is straightforward, the determination of the solar constant is mostly hindered by data. The pioneering work on the measurement of the solar constant was made by Samuel Pierpont Langley, who invented a precise bolometer in 1880. Spaceborne TSI observations, on the other hand, started from 1978, and are made by various instruments covering different time periods, but with some overlaps. Since the disagreement among various instruments is often non-negligible, it is generally difficult to know which instrument is trustworthy during which period. The most recent advance on this topic is put forward by Gueymard (2018), who performed an re-evaluation of the solar constant based on 42 years of TSI data. The conclusion, after careful debate and data analysis, suggests a solar constant of 1361.1 Wm⁻².

A super majority of solar engineering tasks only require the broadband solar irradiance to operate. However, in some PV, photobiological, and photochemical applications, e.g., during characterization of PV materials or determination of photosynthetically active radiation, spectral irradiance is needed. For an all-inclusive compendium of applications of spectral irradiance, the reader is referred to Gueymard (2019). Just like the broadband solar irradiance, spectral irradiance data can be either measured or modeled. Given the fact that spectroradiometers are costly, spectral irradiance models are in high demand. At the moment, the Simple Model of the Atmospheric Radiative Transfer of Sunshine (SMARTS) model is arguably the most popular choice; its 25-year journey has been recently compiled by its inventor (Gueymard, 2019). One should note, however, that most spectral irradiance models, including SMARTS, are restricted to producing clear-sky spectra. In that, the accuracy of these models is highly dependent on the quality of inputs, such as aerosol, water vapor, or surface albedo.

Clear-sky irradiance refers to the irradiance obtained under a cloud-free atmosphere^②. The ratio between the irradiance (or PV power) and its clear-sky expectation is known as the clear-sky index. If not specified, clear-sky index refers to clear-sky index calculated with GHI. The models which are used to estimate the clear-sky irradiance are called clear-sky models, and they can be either physics-based or empirical. Physics-based clear-sky models leverage radiative transfer. However, owing to legacy reasons, simplified radiative transfer or look-up table are ordinarily used by the solar energy community. On the other hand, empirical clear-sky models fit irradiance observations under clear-sky conditions to some mathematical function forms, but model parameters obtained through this way are rarely general. When properly modeled, clear-sky irradiance describes the diurnal and seasonal variation in irradiance caused by all factors but clouds. Therefore, these models are very useful in terms of detrending the irradiance or PV power time series, during forecasting, satellite-to-irradiance modeling, among other data analyses pertaining to radiation modeling. A pair of recent reviews compared 75 clear-sky models for GHI (Sun et al., 2019) and 95 clear-sky models for BNI and DHI (Sun et al., 2021). It was found that the REST2 model (Gueymard, 2008) has the best overall performance, if high-quality input variables are available. REST2 is a physics-based method that takes the effects of aerosol and other atmospheric particulates, which attenuate incoming solar radaition, into consideration. It requires several input variables, included aerosol optical depth and column ozone, which need to be sourced from reanalysis or remote-sensing databases, such as the aforementioned MERRA-2.

Also known as decomposition modeling, separation modeling aims at splitting the beam and diffuse radiation components from the global one, using parameters that are calculable, such as zenith angle or clear-sky index. More specifically, diffuse fraction (the ratio between DHI and GHI) is expressed as a function of clearness index (the ratio between GHI and extraterrestial GHI) and other variables, see Fig. 3. for an example scatter plot. The reason why solar engineers are interested in separation modeling is two-fold. Firstly, when estimating GTI, both GHI and DHI are required. If DHI measurements are unavailable, it needs to be estimated from GHI through separation modeling. Secondly, satellite-to-irradiance algorithms only estimate GHI, whereas the diffuse and beam components in most satellite-derived irradiance products need to be, and in fact are, generated using separation models. The number of separation models in the literature is numerous: there have been more than 150 models proposed so far. Besides one or two exceptions, most of these models are empirical in nature, and the performance of these models depends on the data that is used during model fitting. It is to this effect that the ranking of separation models has been controversial, until the recent review by Gueymard and Ruiz-Arias (2016), who compared 140 separation models using data from worldwide locations. In that review, the Engerer2 model (Engerer, 2015) has been identified as quasi-universal. Since then, many other models have been proposed, and Engerer2 is often used as a benchmark. Among these new proposals, the Yang2 model (Yang and Boland, 2019; Yang, 2021) is the best separation model to date, which is able to outperform Engerer2 by a significant margin (see Yang and Gueymard, 2020, for review), which is due to the fact that Yang2 uses a low-frequency diffuse fraction estimate as input that is able to capture the low-frequency variability in high-frequency diffuse fraction—Yang2 is equivalent of cascading two Engerer2 at different temporal resolutions. Traditional separation models are deterministic, in that, they do not carry any notion of uncertainty on the diffuse fraction estimates. Nonetheless, recent advances in radiation modeling have led to probabilistic separation models, albeit their widespread acceptance has not been fully evident (see Yang and Gueymard, 2020).

Figure 3.  A scatter plot of diffuse fraction $ K $ versus clearness index $ K_T $. The color bar indicates the number of points in the neighborhood. Data source: Fort Peck station, Baseline Surface Radiation Network.

Transposition models convert irradiance components on a horizontal surface to those on tilted surfaces. Since PV panels, when not restricted by topography or installation surface, should be placed on an Equator-facing surface that has a tilt angle comparable to the site's latitude (this is to maximize the annual energy production), transposition modeling is essential in PV design, simulation, and performance evaluation. The number of available transposition models is not as many as separation models, yet, there are about 25 of them. All of those models only differ in the treatment of the diffuse transposition factor, whereas the treatment for direct and reflected radiation components is shared across all models. As reported by Yang (2016), after a worldwide comparison using 18 datasets, the 1990 version of the Perez model (Perez et al., 1990) stood out from a pool of 26 models, and was found to be the most accurate model to date. The Perez model separates the sky dome into three geometrical parts, within each of which the radiance is constant: (1) the circumsolar disk, (2) the horizon band, and (3) an isotropic background. Indeed, based on a bibliometric analysis, the paper by Perez et al. (1990) is the most cited one in the history of the Solar Energy journal, which suggests the popularity of the model. Similar to the case of separation modeling, transposition has been viewed as a deterministic process, historically. Nevertheless, in the recent article by Quan and Yang (2020), probabilistic transposition models have been discussed, and two general strategies for converting traditional deterministic transposition models to probabilistic ones were explained.

Figure 4.  Angular distribution of sky luminances for each of the 15 CIE standard sky types. A solar zenith angle of 60° and a solar azimuth angle of 180° is assumed.

The three-part geometrical framework of the Perez model facilitates the determination of the sky-view factor, which is the part of the sky as seen by the tilted surface. However, in urban environment or terrain with a humpy topography, where obstruction throughout a year is complex and heterogeneous, this simple three-part division of the sky dome is no longer sufficient. Instead, one requires a more granular breakdown of the sky dome—the angular distribution of radiance—to estimate the diffuse irradiance received by a tilted surface. Beside radiance distribution, a similar concept is the angular distribution of luminance, which is essential for daylighting, as required during architectural design. Figure 4 shows the angular distribution of luminance for 15 standard sky types defined by the International Commission on Illumination (CIE, 2004). Models proposed for angular distribution of radiance/luminance are mostly empirical (see Torres and Torres, 2008, for a review), in that, the model coefficients are determined by fitting the model to measured data. The most common type of instrument measuring radiance/luminance is the sky scanner, which sequentially scans the sky dome in a patch-by-patch fashion, every few minutes.

Both satellite-derived irradiance and NWP output are often found biased. To enhance the bankability of a solar energy project, may it be a PV one or a CSP one, the bias in raw data ought to be corrected—this procedure is known as site adaptation in this field, which is similar to the measure–correlate–predict procedure in wind engineering. This is, to some extent, closely related to data assimilation in NWP, in which the output field of NWP is taken as the initial state, and is adjusted by measurements using mathematical method, such as optimal interpolation or Kalman filtering. The idea central to site adaptation is to use a short period (such as one year) of high-accuracy ground-based measurements to correct a long period (typically 10–20 years) of low-accuracy gridded data at a site collocated with the ground-based measurements. Traditionally, site adaptation has always been viewed as a deterministic regression problem, in that, various techniques have been reviewed by Polo et al. (2020). However, as uncertainty quantification slowly gains attention, probabilistic site-adaptation techniques become relevant, which are reviewed and validated by Yang and Gueymard (2021). One should note that site adaptation is a step that cannot be circumvented during solar energy system feasibility study, design, and simulation. For an overview on solar project financing and bankability, the reader is referred to Yang and Liu (2020).

It is customary to regard the two closely interconnected fields, namely, solar irradiance forecasting and solar power forecasting, as one—solar forecasting. At this stage, two common misconceptions must be warned. One of those is the misconception that satisfactory solar power forecasts can be produced using purely data-driven methods. As machine learning and statistical methods become widely available in software packages and tools, many novices tend to simply pass data to these packages and tools without considering the underlying physics; insofar as the methods have not been used previously, novelty can be claimed. This is in fact a very bad practice, and we shall explain why shortly after, in section 3.1. The other common misconception is that the novelty in solar forecasting should be solely revolved around forecasting methodology. Indeed, forecasting methodology is an important aspect, but it is never the only one. The field of solar forecasting has five main aspects: (1) forecasting methodology, (2) post-processing, (3) irradiance-to-power conversion, (4) verification, and (5) materialization of values. These are discussed in Section 3.2.

It is not possible to produce good solar power forecasts without good solar irradiance forecasts. The rationale behind this argument is rather straightforward: solar power, may it be from PV or CSP, varies chiefly according to irradiance condition, in that, there does not seem to be any valid reason why solar irradiance forecasts should not be involved during solar power forecasting. Moreover, because solar irradiance is an atmospheric process, it is best forecast using physical approaches. There have been numerous works that adopt a purely data-driven approach to forecast solar power, in which historical weather and power data are used as input of machine learning and statistical methods. Forecasts generated in this way would be perpetually suboptimal, because they ignores the salient features of solar irradiance. In other words, what distinguishes solar forecasting from any other forecasting domain is not how sophisticated the data-driven methods are, instead, it is the consideration of atmospheric physics, namely, domain knowledge, that matters.

The first and foremost salient characteristic of solar forecasting is reproducibility. As methods and algorithms grow more and more complex, and their number increases exponentially, it would be very time consuming to reproduce the results of forecasting papers of interest without computer code. Statisticians are generous sharers in that respect, since publishing computer code alongside with research articles has been a common practice in their domain. Solar forecasting research, in the early days, was non-reproducible, in that, the code and data were kept proprietary. However, the situation has changed drastically now, owing to a few initiatives by field leaders (e.g., see Yang et al., 2018a). In fact, this switching of publishing regime is no coincidence, for the pitfalls of non-reproducible forecasting research have been debated elsewhere, by general-purpose forecasting experts, in the top journal of their field (Makridakis et al., 2018; Boylan et al., 2015). Moving forward, it is thought that the attention paid to non-reproducible research (or, papers do not present computer code) is going to become less and less—we do not have all day to make suppositions on the technical ambiguities, which exist almost surely in any research paper nowadays.

It is well known that NWP forecasts, when needed, can be issued at arbitrary temporal resolutions. However, most operational NWP forecasts have an hourly resolution, since otherwise, the data to be disseminated would be too big to be practical. This fact poses some challenges to solar forecasters, since the spatial and temporal scale of forecasts required for grid integration is often smaller than the synoptic- or meso-scale NWP output. Hence, to convert the raw NWP forecasts to a form that is readily usable by grid operators, downscaling and other post-processing techniques are needed. Another important aspect of operational forecasting is lead time. It is customary for researchers to design rolling forecast experiments of various sorts in academic papers.^③ Notwithstanding, in an operational sense, rolling forecasts are coupled with lead times. This fact has been largely ignored by solar forecasters. Since longer horizon is more difficult to forecast, the inclusion of lead time is of great importance for mimicking the actual operational forecasting context.

The third salient characteristic of solar forecasting is consisted in its physical nature. Unlike forecasting in a social setting (see Makridakis et al., 2020, for a review on that), forecasting of solar ought to pay more attention on the spatio-temporal behavior of irradiance. Information describing such behavior can be captured by sky cameras, weather satellites, or NWPs, which constitute the three most commonly used forms of exogenous data in solar forecasting. Historically, as first argued by Inman et al. (2013), sky camera, satellite, and NWP data are most suitable for intra-hour, intra-day, and day-ahead solar forecasting, respectively. Nevertheless, with the advent of high-resolution remote sensing and NWP modeling, this correspondence is no longer valid, e.g., 5-min–2-km National Solar Radiation Data Base^③ is well suited for intra-hour forecasting, or the National Oceanic and Atmospheric Administration's High-Resolution Rapid Refresh^③ updates hourly and is thus suitable for intra-day forecasting. In one way or another, utilizing physics-based approaches, as to capture the dynamics of clouds, is what distinguishes solar forecasting from any other forecasting setting.

Weather forecasters are fully aware of the advantages of ensemble forecasting, which issues several equally-likely trajectories of atmospheric processes, rather than just one. An ensemble forecast is a special form of probabilistic forecast, which describes the uncertainty involved in the quantity to be forecast. For probabilistic forecasting in a solar context, the review by van der Meer et al. (2018) is recommended. Reviews on probabilistic weather forecasting are numerous, and the ones by Gneiting and Katzfuss (2014); Gneiting and Raftery (2005) are among the high-quality ones. An ensemble can be constructed in different ways, a typology can be found in Roulston and Smith (2003). In the field of statistics, ensemble forecasting is known as combining forecasts, which has been widely recognized as the best forecasting practice—references and reviews on this topic are too many to list, but a few early reviews are herein enumerated (Wallis, 2011; Armstrong, 2001a; Clemen, 1989). Whereas a majority of existing works combine deterministic forecasts, combining probabilistic forecasts is now trending (Winkler et al., 2019; Clements and Harvey, 2011).

Suppose there is a world without clouds, solar forecasting would be a very straightforward calculation. This is because physics-based clear-sky models are able to attain highly accurate estimates of surface irradiance. Thus, as also mentioned earlier, clear-sky models are often employed by solar forecasters as a detrending tool, such that the variation in clear-sky index can be mostly attributed to the effects of moving and varying clouds. This argument aligns with the principle of forecasting outlined by Armstrong (2001b), which states that when seasonal component is present in the time series, it needs to be removed before forecasting. To that end, if one is to quantify the skill of a particular forecasting method relative to a reference method, forecasts must be first produced in terms of clear-sky index, and then back-transformed to irradiance or solar power for verification and skill computation. Otherwise, the comparison would not be fair, and the resultant skill would be futile. Most solar forecasting papers in the current literature fall victim to this pitfall, in that the reference models of poor performance are often chosen to exaggerate the skill of the proposed models. The recommended forecast verification procedures are discussed in the next section.

What we have just described, namely, reproducibility, operational, physics-based, ensemble and skill, take the acronym of ROPES, which has been recommended by Yang (2019) as the guideline to good solar forecasting research practice. Indeed, the ROPES guideline summarizes all salient characteristics of solar forecasting well. In what follows, another viewpoint on solar forecasting is presented. Instead of examining the salient characteristics, different aspects of solar forecasting, that is, the research topics, are reviewed.

Most well-cited reviews on solar forecasting, such as Antonanzas et al. (2016) or Voyant et al. (2017), have been focusing on enumerating methods. In that, a classification of forecasting techniques seems to have become an essential component of any review paper. Nevertheless, the strategy of classifying forecasting techniques has two main drawbacks. Firstly, the classification of some techniques often depends on individual viewpoints, e.g., regression can be classified into both statistics and machine learning. In other words, the border between one class of methods and another is often fuzzy, and thus rarely allows a clean-cut classification, which may in turn render the classification subjective. Secondly, the enumeration of methods would never be exhaustive. A typical review would contain at most a few hundred references, whereas the number of publications is on the order of thousands, which implies that the strategy of enumeration would always be incomplete. It is often seen sentences like "machine learning methods include A, B and C" in review papers. But why A, B, and C should be mentioned instead of D, E, and F is seldom justified. One possible defense may be that A, B, and C are more popular than D, E, and F. But if this is indeed the motivation, a more objective approach must be sought. In this regard, text mining as a bibliometric tool is useful; the reader is referred to Yang et al. (2018b) for a text-mining-based review on solar forecasting, in which most frequently appeared keywords and topics are extracted from a pool of 1000 recent papers.

Physics-based solar forecasting depends mainly on three types of exogenous data, they are, images captured by sky cameras, remote-sensed data by weather satellite, and output of NWP. In an early review by Inman et al. (2013), the fundamentals of physics-based solar forecasting methods have been described. However, owing to the incomplete understanding on the physical processes and the limited precision of input data, forecasts from the physics-based methods are often found to be biased or uncalibrated (or incorrectly dispersed). Post-processing of initial forecasts is hence often found to be able to improve the quality of forecasts substantially. Another important motivation for post-processing forecasts is to enable bidirectional conversion from deterministic to probabilistic forecasts, which is often required in practice. Similar to the case of base forecasting methods, there have been numerous post-processing methods proposed in the literature, and enumeration of methods would again be inefficient. Therefore, Yang and van der Meer (2021) have advocated the use of thinking tools, which can be thought of as “style” or “mechanism” of post-processing. For example, regression as a thinking tool aims to remove bias from the initial deterministic forecasts, in that, the particular choice of method does not matter, e.g., one can choose either a neural network or a polynomial regression, which, when properly set up and trained, would most likely result in similar outcomes. Other notable recent compendia on post-processing of forecasts include those from Vannitsem et al. (2021, 2018).

The mapping between irradiance and other weather variable to solar power is closely analogous to that between wind speed and wind power; one can refer to such mapping as solar power curve. To convert irradiance to solar power, one can adopt either a direct (or data-driven) approach or an indirect (physical) one. The direct approach views the output of a solar energy system as the dependent variable (or the predictand), and irradiance and other weather variables as the independent variables (or the predictors). In this way, a regressive relationship can be established. In contrast, the indirect approach considers explicitly the physics of different steps of the conversion, which include solar positioning, separation modeling, transposition modeling, PV cell temperature modeling, soiling, shading, mismatch, degradation, among others. Since these models need to be applied sequentially, where the output of one model is the input of the next, the indirect approach is also known as model chain. The model chain concept is well described in the documentation of the pvlib library in Python (Holmgren et al., 2018). Despite the a priori advantage of model chain, its uptake has been stagnant, perhaps due to the complexity and effort required to master and use model chain—recall that each type of radiation model presents numerous choices. One exception is the paper by Mayer and Gróf (2021), who compared several model chains, which is likely to be the most comprehensive paper on irradiance-to-power conversion by far, and thus is highly recommended.

After a solar irradiance or power forecast is issued, it is customary to validate it, so that actions can be opted as to improve the next forecast. Good is an abstract notion, but so long as the question "what constitutes a good solar forecast" is concerned, the essay by Murphy (1993) is widely recognized as seminal, in that, consistency, quality, and value have been argued to jointly characterize the most desirable weather forecasts. Forecast verification has hitherto been a major focus of meteorology, and atmospheric scientists are expected to be familiar with, or at least aware of, the intricacy and technical depth of this topic. Whereas the book by Jolliffe and Stephenson (2012) provides an overview on forecast verification, reviews on more specific topics, such as spatial forecast verification (Gilleland et al., 2010), consistency between judgment and forecast (Gneiting, 2011), accuracy measures for deterministic forecasts (Hyndman and Koehler, 2006), are also available. Particularly worth mentioning is a pair of recent reviews, which details the verification of solar forecasts, one for deterministic forecasts (Yang et al., 2020) and the other probabilistic (Lauret et al., 2019).

By "materialization of values", we mean quantifying the usefulness of forecasts. Since a forecast has no intrinsic value, its value must be materialized through its ability to influence decision-making (Murphy, 1993). At present, nor is there any work reviewing the relationship between the quality and value of solar forecasts. This is, to a large extent, due to the fact that different countries have different grid codes, under which the monetary compensation and penalty are advised; if one is to compare the economic values of two sets of forecasts in different markets, it would not be fair. In any case, the standards and challenges of using solar forecasts for grid-side operations, as described by Yang et al. (2021), are often shared by power grids worldwide. In parallel, Li and Zhang (2020); Ahmed and Khalid (2019) presented reviews on several concrete examples of how probabilistic forecasts may be used in power system operations, although in reality, given the conservative nature of the power system industry, probabilistic forecasts have yet to receive any sizable adoption. With the increasing penetration of solar energy, the value of solar forecasts would become more apparent. In the paper by Makarov et al. (2011), the basic operational practices of the California Independent System Operator (CAISO) are reviewed, from which one can understand how better forecasts can lead to higher load-following performance, less regulation and reserves, and thus higher economic values. Other notable reviews on the materialization of values of solar forecasts also exist (e.g., Emmanuel et al., 2020; Sampath Kumar et al., 2020).

In this überreview, topics related to solar resource assessment and forecasting have been listed concisely. This überreview can thus act as a catalog of research, for anyone who wishes to enter the field or stay up-to-date. Generally speaking, solar resource assessment mainly focuses on data and models. In that, ground-based, remote-sensed, and dynamically modeled radiation data play a vital part during solar energy system design, simulation, and performance evaluation. Additionally, the choice of radiation models of various sorts, such as transposition model, separation model, or site-adaptation model, also should not be viewed as trivial. Much effort has been devoted to the development and validation of these radiation models. Hence, opting the recommended radiation models during research is highly advised.

On the other hand, forecasting of solar irradiance, as a means to forecast solar power, is required during grid integration. In this regard, the value of solar forecasts solely resides in their ability to influence decision-making pertaining to power system operations. Solar forecasting research is never about which model is fancier and how much machine learning is used. Instead, understanding the salient features of solar irradiance should be emphasized. Raw solar forecasts should be generated using physics-based methods, and post-processed using statistical and machine-learning models. Moreover, irradiance-to-power conversion and forecast verification have yet to receive the attention they deserve.

In any case, solar resource assessment and forecasting can benefit from the participation of atmospheric scientists. Cloud is the critical parameter that ties atmospheric science and solar energy together. Among all cloud types, the largest uncertainty is associated with cumulus and cirrus. Further understanding of the dynamic and physical processes governing the production, maintenance, and disappearing of these clouds is necessary, as to improve the forecast skill of NWP, which would then certainly advance solar resource assessment and forecasting. Aerosol is another parameter that is key to solar engineering. Given the fact that many large solar energy systems are located in semi-arid regions, they are often affected by dust episodes. Progress in atmospheric chemistry would enhance our understanding of how natural and anthropogenic aerosols impact solar energy resources, and how to consider these effects in NWP. In this regard, developments in atmospheric science would be an important booster for solar engineering. To achieve this goal, mutual understanding is very important. First, the solar energy community should be familiar with the latest developments in atmospheric sciences. Second, the atmospheric sciences community should know the needs of solar engineers. Indeed, if we are to achieve carbon neutrality by mid of this century, a high proportion of solar energy in the energy mix is a must, and for that, interdisciplinary collaboration is absolutely vital. We should wish this review can help bridge the two communities.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.