Big data: The energy revolution no one is talking about
The big data revolution is already changing many industries in America. Companies are leveraging analytics and artificial intelligence (AI) to harvest massive amounts of data and turning it into insights that create new business models and customer acquisition strategies. In turn, the big data analytics insights provide competitive advantages, driving America’s most successful businesses to stratospheric market valuations.
Surprisingly, there is less focus on the transformative potential of big data in the energy sector. Big technology companies, such as Google and Microsoft, have created tools to enable 24/7 clean energy supply and expand cloud-technology services. But electric companies, particularly in the U.S., have a long way to go toward fully harnessing the power of big data.
Recently, New York began testing how much value it can create by expanding energy data access. In doing so, New York may be leading a long-anticipated revolution in how utilities, developers, and customers can access, use, and monetize data to unlock more value. As the New York data access proceeding evolves, it serves as an industry case study for regulators, developers, and utilities around the country.
New York utilities and regulators lead the way in harnessing big data potential
With utility digital transformation underway, regulators and stakeholders in New York believe that access to broader energy data will allow market participants to create incremental value for customers—and support the state in reaching its clean energy goals.
Since the beginning of the Reforming the Energy Vision (REV), the New York Public Service Commission (NY PSC) has maintained that customer and third-party access to energy data are critical to operational efficiency, market innovation, and competition. And in delivering that message, NY PSC pushed the development of many state data access initiatives, such as green button connect (GBC), building benchmarking, Utility Energy Registry (UER), hosting capacity maps, and system data portals.
In 2021, the NY PSC issued two important data orders—the Integrated Energy Data Resource (IEDR) Order and the Data Access Framework (DAF) Order—that establish new statewide data access standards, platforms, and processes.
The IEDR Order creates a statewide centralized data access platform to feature utility data—customer and system data—and other energy-related data (e.g., EV registration, building characteristics, DER operations). The IEDR will transfer big data from data custodians—utilities and other parties—to a centralized platform and will use data analytics to give users information to support their businesses’ needs based on defined data use cases. New York State’s largest investor-owned utility, Con Edison, supports the IEDR effort.
"Customers need information to meet their evolving energy needs whether that be pursuing energy efficiency, replacing their boiler with a heat pump or purchasing an electric vehicle. Con Edison recognizes the importance of providing customers and third parties with actionable data to inform these decisions and to help speed the deployment of clean energy solutions. We are excited to be completing our Advanced Meter Infrastructure (AMI) and recognize this will be foundational to realizing that clean energy future."
— Damian Sciano, Director of Statewide Energy Data Integration, Con Edison
The DAF Order addresses the operational and regulatory challenges related to grid security and customer privacy as data access expands in scope. The DAF explores and determines appropriate cybersecurity and privacy protections, data access mechanisms, data quality standards, and customer consent processes. The DAF Order centralizes the certification process for third parties seeking energy data access and requires utilities to submit several regulatory filings that describe existing processes and propose plans to implement the framework.
Why utilities are going big on data
Utilities have begun a major grid and digital transformation, giving them access to more granular and temporally specific data than ever before. A traditional utility meter can read a customer’s monthly energy usage. But smart meters—already at around 75% penetration in the U.S. and expected to hit 80% by end of 2021—can take readings in 5-to-15-minute intervals. That means in any given year, a utility customer with a smart meter can access 35,040 (i.e., 365 days x 24 hours x 4 intervals) energy-usage data points—instead of the previously available 12.
In addition to smart meter data, other grid and customer devices are now able to collect more significant data. There are clear opportunities embedded in energy-related data: for example, insights into customer usage patterns, energy savings, Distributed Energy Resources (DER) optimization, and customer acquisition.
But there are challenges to navigate. For investor-owned utilities (IOUs) with millions of customers, measuring, collecting, storing, and sharing enormous amounts of data can quickly become a daunting task. Utilities need to resolve technical issues in tandem with business considerations to identify what they can do, how they can do it, how much it will cost, and what value they can create.
Some examples of data-driven monetized value include the existing utility demand side management (DSM) programs, especially demand response (DR) and pay for performance (P4P), time varying rates (TVR), and managed electric vehicle (EV) or battery storage charging. Data access has also been instrumental in delivering positive societal benefits in these customer programs. For utilities experiencing or expecting high DER penetration, granular data access, strong data analytics, and proper distribution control capabilities will be critical for aggregating and orchestrating these resources to maximize grid flexibility and continue to deliver reliable and affordable service. But even for those that aren’t, the new data landscape will be transformative, and it is here to stay.
There are also significant privacy and cybersecurity risks when opening data access to third parties. Energy-usage data—especially when combined with other commercially available data sources—can reveal a lot about a customer. Sharing that data with third parties requires strict protections around who can access it, how it is used, and how it is protected. Negotiations over these standards can be tense, as observed in the New York proceedings.
A new spotlight on rate cases, grid mod plans, and planning from regulators
Many existing regulatory data access requirements originated during AMI, grid modernization, and distribution system planning (DSP) proceedings. As utilities continue to invest in and deploy new grid technologies (i.e., AMI, customer portals, grid sensors, DER) that collect advanced data types and formats, stakeholders and regulators will demand more transparency for customers, third parties, and the public. As the market grows more attuned to the value of energy data access, questions such as how data will be used and where value can be captured will be front and center in rate cases and other regulatory proceedings such as distribution and grid modernization plans.
In other words, the market is entering a new phase of utilities introducing technology investment plans that require a clear articulation of potential benefits grounded in data use cases. At the same time, a major challenge for many utilities will likely be that stakeholder and regulator expectations may outpace their capabilities. If utilities lack the functionalities to support new data access requirements, they will need to seek incremental investments and budgets, introducing the question of funding and cost allocation.
In addition, increased energy data access may bring market innovation, but it also creates new privacy and cybersecurity risks. As states continue to adopt new data privacy and security legislations and regulations, sharing customer and system data will become a more complicated and onerous process with additional disclosures, privacy, and safety requirements for all the parties involved. Also adding to the list of investments and regulatory review.
The table below includes considerations from the perspective of key energy data access players.
How other states are acting on data access
Although no state has progressed as far—or as fast—as New York, some are actively exploring data access. In 2021, many states engaged in data access discussion under the promise of creating more transparency, enabling more dynamic energy markets, offering innovative customer products and services, and supporting clean energy goals.
New Hampshire’s stakeholders reached a settlement agreement to move forward with exploring a statewide energy data platform and Maine recently passed a bill to evaluate its feasibility. In Minnesota, the IOUs submitted their first annual open data access compliance report and are participating in parallel stakeholder discussions to explore the risk and benefits of sharing system data with third parties through hosting capacity maps. In addition, New Jersey, Michigan, Maryland, the District of Columbia, and Connecticut have active stakeholder engagement processes to explore access to customer and/or system data.
Conclusion: The future of energy data is quickly approaching
The energy data revolution is underway, and data access mechanisms can originate from utilities, government agencies, DER providers, aggregators, or a combination of those. Deciding on the data access delivery model should consider best value (i.e., timing, quality, costs) and privacy and security risk management.
We are already observing rapid change on short timescales, and it’s not hard to imagine even more profound data-driven innovations in utility business models and the energy market in the near future. Some possibilities include:
- Big data analytics companies monetize new services based on energy-related data
- New customer products or services—in energy and beyond
- New expectations for utility business models, including partnerships and offerings based on their ability to provide energy data
The market is moving on energy data access, regulators and stakeholders are taking notice, and developments in New York over the next 12 months will provide valuable lessons for other states. We should all pay attention.
