Evaluating our model's ability to measure solar project performance
Accurate and reliable photovoltaic (PV) generation estimates are critical for the development, investment, and operation of solar facilities. Yet lately industry publications have been cataloging potential underperformance within the U.S. solar PV fleet.
Project stakeholders have to be confident—during development and through operation—in P50 and P99 generation estimates in order to appropriately understand project risks. While industry practices for production modeling have improved over the past decade, the rapid evolution of the solar PV industry requires ongoing evaluation of best practices to ensure project stakeholders can make informed decisions.
As part of our ongoing commitment to deliver best-in-class production estimates, we’ve undertaken efforts to validate our internal practices and procedures for solar PV generation modeling. We utilize continual improvement methods to provide confidence for parties involved in project transactions and operations.
Methodology
Operating solar PV facilities must be examined for an appropriate review of solar PV generation modeling practices. We compared actual, historical generation to our estimates of generation at completion of construction. Our goal was to compare our forecasted energy production to the actual production—absent material impact of system unavailability and degradation—in order to provide a validation of our modeling assumptions and procedures.
We recognize that long-term degradation and operational losses can significantly impact performance. However, such losses are separate from the specific intent to validate our generation modeling practices. Therefore, we sought to identify facilities with long periods of operation without significant downtime. We have also chosen to focus our review on the first year of commercial operations in order to reduce the impact of degradation on our assessment.
Facilities review
Five facilities were chosen for this review, a summary of which is found in Table 1. The five facilities were selected based upon factors related to diversity in terms of size, mounting type, module and inverter manufacturer and vintage, and geography.
For each facility, we reviewed site-specific, satellite-based solar resource data developed by Clean Power Research, LLC version 3.4 for the period of review. We also reviewed historic precipitation data from the National Oceanic and Atmospheric Administration (NOAA) weather station nearest to each facility. When reviewing NOAA data, only locations with meaningfully complete records of daily rainfall during the period of review were examined.
In conjunction with the NOAA precipitation data, we assumed a characteristic soiling accumulation rate based on our experience with utility-scale PV facilities in the various regions. For projects in regions that experience snowfall, we estimated monthly generation snow losses at each facility using well-validated snow loss algorithms within the industry, as well as historic snowfall data from the nearest National Operational Hydrologic Remote Sensing Center to each facility with a meaningfully complete record of daily snowfall during the year examined. We have modeled estimated energy generation for the facilities using version 6.78 of PVsyst Photovoltaic Software (PVsyst). We utilized as-built civil and electrical drawings to inform the assumptions needed to develop generation estimates using PVsyst.
Recent modeling improvements
We reviewed the monthly operations and maintenance reports to determine the availability and actual energy generation in each month. The lost energy associated with availability losses was added back into the actual monthly generation, and the result compared to the generation estimates. For three of the five facilities, we found periods of unreported availability outages. These outages were apparent from the operating data but not identified in the operating reports. Such outages were similarly added back into the actual monthly generation. We reviewed intra-hour irradiance fluctuation—resulting in inverter clipping during short periods—as a contributing loss factor. Commonly referred to as sub-hourly clipping, modeling such impact was not present in our original generation estimates but has since become a standard practice for current generation estimates. Due to the temporal resolution of the PVsyst energy model, which is based on hourly averages, inverter clipping losses resulting from rapid changes from low irradiance to high irradiance during fluctuating cloud coverage may not be fully captured.
This can result in an overestimation of the production of a typical PV system. We’ve developed an internal tool to capture potential inverter clipping losses due to sub-hourly irradiance and found that within this group of facilities the inclusion of sub-hourly clipping losses allowed for closer alignment between the model and the actual performance of the facilities.
Our modeling accuracy
For each project, we calculated the ratio of actual production corrected for 100% availability to modeled production at 100% availability (the Modeling Accuracy Ratio). If an estimate was created that was identical to operational performance, the facility would have a Modeling Accuracy Ratio of 100%. A Modeling Accuracy Ratio below 100% indicates that our model overpredicted the amount of energy generated, while a Modeling Accuracy Ratio above 100% indicates our model underpredicted the energy generation. The annual results are summarized in Table 2.
The results show a mean Modeling Accuracy Ratio of 99.3% with a standard deviation of 2.6%. The mean Modeling Accuracy Ratio indicates that our modelling practices are overpredicting the amount of energy generated by approximately 0.7%. Statistical analysis of the standard deviation performed utilizing a t-Distribution Critical Values Table shows that this mean corresponds with an uncertainty of +7.2% at a 95% confidence interval. This aligns with our team’s understanding of the combined uncertainties associated with PVsyst, satellite-based irradiance data, and modeling input assumptions.
Our typical modeling practice would result in a project-specific uncertainty. However, we did not explicitly calculate for this review. Instead, a review of a representative sample of solar PV generation estimates that we developed for the purpose of financing due diligence in 2020 and 2021 shows a mean uncertainty calculated of +7.0% at a 95% confidence interval, which is generally consistent with the distribution shown in Table 2.
Understanding the main drivers of system underperformance
Based on the facilities review, our modeling practices are capable of achieving strong agreement between actual and modeled production. This is in part due to the inclusion of sub-hourly clipping losses, which have only recently begun to be broadly understood by the industry. The variability of the Modeling Accuracy Ratios aligns well with our typically assumed modeling uncertainty.
During this analysis, we discovered clear evidence of unreported outages at the facilities. Underperformance beyond the small modeling bias was driven mainly by reported and unreported outages, which in some months resulted in losses of up to 30%. Such a delta serves as a critical reminder that in order to achieve estimated production, facilities must be monitored, operated, and managed in a manner that is consistent with industry best practices. To the extent that projects are not operated in such a way, availability losses and operational issues can result in significant underperformance.
We recognize the PV modeling community must monitor and adjust modeling practices to reduce bias and provide valuable production estimates. For our part, we will continue to expand this study to include ongoing research into potential areas of modeling error. Some of these areas include imperfect tracking, improved project-specific soiling considerations, and improved modeling of the impacts of site undulation.
However, our findings point to system unavailability as one of the main drivers of system underperformance, rather than aggressive production modeling assumptions. Understanding reasons for system unavailability and underperformance is of critical importance to close the feedback loop on performance modeling. It’s also important to understand drivers that result in lost production and ultimately lost revenue. Therefore, our team is currently undertaking additional research to further qualify the impacts of system unavailability on long-term PV production, as well as variations across operations and maintenance scope and practices.
