Energy Renewed: Impact of CAPE software on grid protection and reliability

Full transcript below:

Katie: Welcome to "Energy Renewed'' a podcast by ICF, the meeting of the minds and renewable energy, where people come together to discuss ideas and synergies to propel the industry forward. I'm Katie Janik from ICF and the host of ''Energy Renewed." ICF provides technical advisory services to lenders, investors, and project owners for renewable energy technologies and processes. In this podcast series, we will consider varying viewpoints ranging from policy topics to equipment components. Welcome, in this episode, we are discussing CAPE software with Paul McGuire, one of the founders of CAPE software and with Baldwin Yeung, partner and founder of CMY Solutions. More than 50% of utilities use CAPE software to do simulations, short circuit programs, and protection. We are excited to have Paul here as he is a pioneer in the power industry, as it relates to software and grid reliability. And we are equally excited to hear Baldwin's thoughts and discuss the relevance of reliability as we bring more renewables onto the grid and experienced rolling grid outages in some parts of the country. Baldwin, will you take a moment and introduce yourself?

Baldwin: Sure. Thank you, Katie. I'm Baldwin Yeung of CMY Solutions and just quickly I've just been in the industry working on protection software, facility software, NERC compliance, and one of the interesting pieces that our role that's evolved into is the data management of all of this, the John Rove software and the importance that it's gonna be to us in the future here. So thank you for having me. Katie: Thanks for being here. And Paul, will you take a moment and introduce yourself? 

Paul: Thank you, Katie. I would be happy to. Yes. My name is Paul McGuire. I have been with a small company in Ann Arbor, Michigan now known or formerly known as Electric Con International, who developed software to analyze the power system grid specifically on the areas of power system protection. And we can explain more about that later. I am now retired or very recently retired and had sold the company to Siemens in the United States two years ago in order to allow the company and its valuable contributions to our profession to carry on, and that has now been done. So with that, I guess I will say thank you for having me today and I hope that I can provide information that's useful to you.

The background story of CAPE software

Katie: Oh, I'm sure you will. So perhaps let's start on the background of the software. I'm curious as to what was the genesis of the software? How did it come to be?

Paul: It's a very interesting story. The company that I was part of had been working since the early 1970s on developing computer programs for power system analysis, not in the area of protection where we ended up, but in the areas of transmission simulation, primarily. We developed a short circuit program on our own knowing that there would be some need for that function that no one else had really done well.

We did that over the time, 1982, 1983 and 1984. It still hadn't gotten into the topic of system protection. We had sold that software on personal computers by the way, which was a big deal and something very new in 1984, but we had software that was running on single and double diskette drive personal computers, no hard drives. That was seen by Georgia Power Company, a member of the Southern company.

And they realized the need for software that was support the system protection function. And short circuit is a key component of software that would do that sort of support, but that had never been developed at that point. And they came to us because they saw how well the short circuit program worked and asked would we be willing to develop with R&D function monies.

The product that they had come up with a name for Computer Added Protection Engineering or CAPE for short and nine of its functions, actually eight of its functions. And we added one immediately and that was a database management system would be needed to manage all the data that's involved. So it is a textbook story that the customer came to us and we realized the value of, and the potential for what they were asking us to develop and realized that we would need more money than one company was going to provide.

We convinced them to allow us to go out and find nine other companies would support financially this development, but not just financially, also with the technical guidance, suggestions, participation. In other words, I know it's going to sound funny, but here's a very important program consisting of many, many components today that was developed by committee. And I hope you guys see the humor in that statement, but in fact, it worked. Engineers want to do a good job.

They want to do a job and do it well. And they were only too willing to participate with us every three months in two-day meetings where we would discuss the product that was being developed and how it should be done, what it should look like. And that went on for now four or five years until we had gotten a good start on the development, development that one of us thought would take only 18 months before we started the work. Others of us thought it would take a little bit longer, but in truth it took about 20 years to get to the point of completing the work that was originally envisioned and adding on to that.

The CAPE software team

Katie: And let's talk about just before we move on, let's talk about the team that you had. So when you're talking about you're using terms like us and we, so you had a team of people you were working with that already had the short circuit program in place. And then that was what Southern power company wanted to expand upon. Can you talk a little bit about the team that you were working with at the time?

Paul: I'd be happy to. The miracle of it all was that we only went out and searched for one additional person beyond the people we already had. Two of us had been classmates at the University of Michigan in the graduate program in power system engineering there. Another was in from England, actually Scotland and his specialty was applied mathematics, not power systems analysis, but became a critical component of the team.

And they were only others that we gathered along the way. I would say we ended up developing the program with an average of about six people, five to six people over a period from 1985 when the development actually began or the discussion began and then develop a year or two later development a year or two later through really through the present time. But through the core of the development, that group of people I would say that they consisted of generally Ph.D., recent Ph.D. students who come from all over.

We have people on the team now from Mexico. One person I'm thinking of was educated in Northern Canada if you can imagine that contrast. Three people from various parts of China, two from India and then some people who were actually born in the United States such as myself. So it was really a coalition of participants that we picked up bit by bit as we had room on the staff for them, but who surprisingly had talents that were just the right ones that we needed for the work. Some, we didn't realize had it until we get into the project a ways, but it was amazing that just the right people came together to accomplish this. Katie, did you have any other aspect of the team?

Baldwin: Paul, I had a question actually, two questions, really. The first being, would you say, was it mostly a democracy while you were putting this together? Or is it more there was a common goal where everyone had input, but at the end of the day, somebody had to be the, what was gonna make the final cut of the software?

Paul: Well, we certainly had a manager of the project that was me. And so, but I would still say that there was a strong element of democracy in there in that if you're a good leader and manager or you're listening to ideas from everyone and making them understand that their values matter, that their suggestions matter, and utilize them in the right combination. So that I think was the proper role that I played in doing that. And not only me.

First of all, when the project started the professor who was the president of the company at that time, Dr. Mark Ens was in that position, but I was the engineering manager and making those decisions, but never single-handedly, always in discussion. So it was very much a group sort of thing. Furthermore, I mentioned that we had 10 utilities who sent one or two attendees to every advisory committee meeting that we held, which was again, every three months for two days usually to discuss the design and develop the ideas, then we would go away and work on those. Do some of the programming, come back three months later and demonstrate what we had done so far that led to more communication and more progress in the thinking of how this product would look.

But I would say that when we began, we had a functional specification. In other words, what major components do we want this product to have, which didn't change. We expanded on it a little bit as time went on, but it didn't change. How we were to do it, that's something we had to do as we went along where we, again, had ideas based on work we had already done the previous 15 years. In other areas of power system analysis, we could apply some of those ideas again, but no, it was an evolution and it was a fortuitous one for all of us that we had the right people with the right attitudes to make contributions, to make it work.

Baldwin: All these people that he's mentioned, if you go into detail, not all of them are power people. So it was like the perfect storm of mathematician, engineer, a database guy. I think really Paul, I think that's understated of the success of CAPE.

Paul: I probably do. We've always been there. We've never tried to boast or anything, only to show people by our actions and the product that we develop that we were the ones to go with.

Katie: Well, to me, it's foresight that you put together a diverse team. You put together a program that could fit on personal computers in the early to mid-80s when not everyone had a personal computer. To me, I feel like this episode could easily be discussed about leadership, right, leadership skills. But from our prior conversations, it sounds like you weren't the first to develop this type of short circuit program for computers. Is that correct? It was just the most flexible.

Managing use cases

Paul: It is correct. There were people, Philadelphia Electric Company (PECO for short) had developed a short circuit program using a technique that was being used was one that did not, it prohibited study of large networks. There are terms for all of this that I won't introduce really, but you basically in getting the calculation would end up with a set of numbers that was so large to process that the calculation would be slow if you got to be more than what we call 300 buses. Those buses are points in a network that are interconnected by transmission lines and transformers.

Well, you couldn't get larger than 300 without running into time problems. Well, the real networks are many thousands of buses even then. Today, 50,000 bus cases, not unheard of by any means. This technique that we developed and simultaneously with a group in California developed without knowledge of each other working on it, we developed a program anyway, that would solve basically any size system, almost instantaneously.

Katie: So in comparison, just to put this in perspective, you mentioned that the existing software at the time would only say it used 300 buses?

Paul: 300 or a little more. More time you wanted to wait for an answer, the bigger system you could look at.

Katie: And now what is the use case?

Paul: There really isn't a limit anymore, but I would say our average customer has a system of about 5,000 to 6,000 buses. But many of them get up to 20,000 buses and some are up in the level of even 50,000 buses. And our vision is that we should be and will need to be in the future, looking at the model of the entire North America.

Katie: Wow. That is extraordinary to go from...Paul: It should be 100,000 buses easily.

Katie: To go from such a small I'm calling them use cases, but to go from such a small use case using 300 buses to 20, 20, 25,000.

Baldwin: I think, that's a real like testament to how mathematical the model is at its core, right? Paul, because the math is what's is basically what's driving it's limitation.

Paul: Mark he's the professor who founded us at Michigan and another professor—well, formerly he's retired too—at the University of Wisconsin who was Mark's best Ph.D. student. Those two working together had the ideas and the vision that we needed to have a program that could handle much larger systems. Now, it turns out when the Southern company folks came to us, they were suggesting at least 2,500 buses, but Mark and Fernando Alvarado knew very well that we needed much to be able to handle a much larger system.

And they designed a technique. They were numerical methods developers, and they designed a technique that would basically give us instantaneous answers anywhere in the system model we wanted, no matter how large the system is. And that was an absolutely necessary component though we didn't know it at the time of the development of CAPE.

The database

Katie: And the start of your database.

Paul: The database was always—and remains today—the core, the foundation of the whole thing.

Baldwin: Tell me Paul, if this is a good analogy of it is that the mathematics of it is kind of like a race car. But in this day and age, there's so much data coming from so many different places trying to drive this car with all the inputs it's kind of like driving on the gravel road if you're not careful. You can do whatever you want it to do, but it's whatever you plan to feed into it. And I think that's the dangers that we see out there. At least as a consultant, that's my danger.

Paul: Well, certainly the data is the essential part of it. And in fact, it must be accurate and must represent what's really out there in order to get, well, you've heard the expression garbage in, garbage out. You want to avoid getting garbage out, so we've got to have a good model. That's why I call the database, the foundation of this whole situation, this whole area of study.

Katie: And because of the size of the database, the number of simulations that you can perform can have up to, what number of different types of conditions or what number of different types of scenarios?

Paul: Yeah. So you might get very good answers from one single simulation, that's fine. And that's done, that's an interactive type of study that's done all the time by the protection engineers who use the product.

But there is a level well above that, where you would like to do a massive review of the entire network, what we call a wide-area coordination review that gets to the subject of protective devices and what they do. You need to have the devices that are closest to a fault that might occur on the network, be the ones that operate and separate out the smallest portion of the network possible to eliminate the fault. And you, sometimes that separation is temporary. Sometimes it's gonna be a longer term. But if you're trying to find all those situations that could happen in your network and want to evaluate all the protection that we have designed into the system to be adequate in all cases that we can think of, now, you might be talking is somewhere in the range of 500 to 1,000 studies per transmission line and you may have even just at the high voltage levels you might have 500 to 1,000 lines to deal with. You could imagine getting a thousand times 1,000 cases to study in order to accomplish that wide-area review of your whole protection system.

That's not something that's done at the blink of an eye. It could take weeks and months to do, and it does. And we know that that's too slow and we need to do it much more of the studies, many more of them and do that much faster. And we are in the process of rewriting the program to make it faster and also utilizing multiple computers simultaneously to accomplish jobs. So that's where the million number could easily come from.

Grid reliability

Katie: And just for our listeners, can you put it into context in the sense of having a million different conditions or scenarios, what it does to contribute to the reliability of the grid?

Paul: Well, I can tell you what our contribution is. To increase the reliability of the grid, one has to have a protection system that will do what I was just saying before. And that is detect the presence of the fault and take an action that has minimal effect on the customers who are using that grid to get their power, their energy from the grid. All right. We don't wanna lose that any more often than we have to. And so we need the protective devices to be, first of all, many of them throughout the system.

Generally, many of them at each end of each transmission line, by the way. Two, we need those devices to operate ahead of devices that you would consider backup, meaning you have a lightning strike on a transmission line, a given line. You want the protection at both ends of the line to operate very quickly and not only just very quickly, but well ahead of backup protection on other lines that would, if they were to open their circuit, breakers would cause more of a network to be lost and more customers potentially to be lost. So you have to worry about that.

Now, a newcomer to this topic would want, as you understand that a network is like a, something like a screen on your screen door of a storm door. In other words, you've got transmission lines that come together at points to satisfy load or to receive generation, to receive power from generators that may exist there. And what's very important again, is to detect the presence of a fault, which usually is voltages and currents that are out of, and way beyond the norm. And it was very low voltage is a very high current and use that as a method to detect that, Hey, there is a problem in that.

Furthermore, one has to have protection that can find the probable location of that fault so that they will operate in a coordinated way, remove the fault with least impact on the network itself, keep the rest of the network running while we are taking out the faulted condition.

Baldwin: I was going to say, I think you're almost underselling capable of it, Paul, because within the utility, there's always competing departments. For example, operations and planning, which is just as important as us protecting the lines, making sure everyone it operates correctly and you're getting the best rates possible. And CAPE itself is usually the foundational piece to operations and planning. Really the tool that most utility is the best in secret, but did they get all the design, they put it into CAPE they get these values out of CAPE clearing or impedances, and then it propagates through a utility as quickly as possible.

So when Paul is discussing here, Hey, these faults are happening or this we're trying to line that up with what's happening in real life. And every day it gets more and more difficult because in the past it was a pretty simple system. It was, you got your power plants, you got your customers, and we're just trying to equal it out. Now with renewables and frequent miles and all these new competing projects and ISOs, this is why this has become so much more important that we ensure these models are correct. It's the basis of basically how we protect our grid.

The impact of renewables

Katie: Let's talk about the impact of renewables. So how does the rise of renewables complicate the simulations or complicate the situation?

Paul: One of the complications is of course, that it's all spread out over the network where before you relied upon a relatively small number of generating stations that were, and because of small, a number of there are not so many of them in the network. It was easier to protect them than it would be today when you have solar and wind and other renewable sources. But those are the two big ones that might be spread out everywhere. And there's another aspect of course, to the renewables and that is that they're not always available. Wind doesn't always blow, the sun doesn't always shine, and certainly quite regularly the sun doesn't shine.

And so you cannot depend on them 24 hours a day, even though power is needed 24 hours a day. Yes, a greater amount is needed in know workday when people are out doing things and using that energy, but that's an issue and there's another one that we need to discuss and that is that the renewables while they're intermittent, they need to have, have there needs to be some form of storage. If they're gonna be used in the large quantities that is large fractions that are governmental authorities and people would like to have them be used. You have to have some way of storing that power because the wind doesn't always blow and the sun doesn't always shine.

And that is a problem that has not been solved. There are various ways of trying to do it that is to store the energy while you have excess somehow, and then utilize that excess energy when you need it which might be when there's a dead calm and everything is cloudy, let's say. And there are hopes for the future, but I don't know exactly where the solution resides entirely things that probably there is probably no one solution to the problem. It's not going to be batteries.

Those are using rare earth materials that I just don't see us having in the quantities that make batteries as a solution. But there is something being worked on now called the hydrogen economy where one could use what's called the electrolysis to that is a direct injection of electricity into water to cause the hydrogen and oxygen H2O, we all know that's water to be separated into individual atoms or molecules, oxygen on one side hydrogen on the other O2 and H2 and then save those, store those in some fashion, at least the hydrogen part and recombine them by effectively burning them to get the energy back when you need it.

The waste products of hydrogen and oxygen are nothing but water, and so it's very attractive, but it hasn't been perfected yet, even though electrolysis has been around for a very, very long time. And the idea we all did it in high school chemistry lab classes. Doing it in a commercially effective way is not been accomplished yet, but is being worked on for sure.

Katie: Hydrogen storage is fascinating. Goldman Sachs has a report out on hydrogen storage and it is quite interesting and it describes what you just outlined. So just to recap, before we get to more solutions, so it sounds like with the rise of renewable, you have a resource issue in terms of having consistent resource that contributes to the reliability of the grid. And then you also have the storage piece and whether or not it's going to be efficient enough or productive enough to contribute to the reliability of the grid.

What’s the best model?

Baldwin: We also had a modeling issue, I would say five years ago, too, that I think Paul and them have ironed out, but that was a big issue with renewables was everybody was when there's a wind generator or solar generator, we didn't know how to put that into the model accurately. And everybody had their version of what it should look like. That was a huge issue between developers and utilities to accurately just because renewables are DC and we have to convert them through power electronics. And the problem is power electronic engineers and power engineers, it's like Greek to each other. And right, Paul, that was a huge headache.

Paul: I wish I could call it all in the past. I'd say it continues to be today. We haven't settled out on what the best models are to use for wind turbine generators, for example. And part of the issue is the competitive nature of that business and manufacturers, vendors, not wanting to share in details about their control systems with people like us who need to model them. So there is an issue right there. We're working on it. And I would say one of the, in North America, at least one of the prime contributors to and pushers of the development of better models of these devices use the Electric Power Research Institute in California. Now, principally in California, we've got offices in many places, but they've done a great job. They've been helping us encouraging us to implement models that they have developed to see how they work, which we have been doing.

That inertia is the point that I'm trying to bring out here is what systems need or depend on to stay stable. When you have wind power in large numbers, they don't have any appreciable inertia. They cannot withstand that sort of jostle or hits. Solar has none. And so getting around that problem in other words, you're losing inertia if you retire these coal plants or other light, certainly nuclear plants for the large generators that they have, have a lot of inertia, a lot of weight behind them. And they can withstand perturbations, but the other devices can't. And so you have to work that into the mix and that's what California is wrestling with right now. And not only them.

Baldwin: Katie, that's the reason why I wanted to highlight that was that somehow becomes our ceiling when we're developing projects, because we're having a struggle in the industry, how much renewable is too much. And it goes back to our original conversation is how good the data is. And if we can't accurately predict what these turbines look like in real time, or how much wind is truly out there, the government's first move is to make sure everyone gets electricity stable. And that becomes a limiting factor for all of us from a compliance standpoint. I just wanted to really highlight that there.

Katie: And we already had seen that, right? This part is what you both are describing. California has seen this with the wildfires. It's also seen this with rolling brownouts or outages.

Paul: Yes. Either caused directly by the fire or as a well, a method of trying to minimize the outbreak of fires because certainly we probably have all seen or heard of a transformer self-destructing somehow and causing a lot of sparks or fault, whatever might cause a fault might be a tree, might be a lightning, but whatever does there'll be sparks associated with that and therefore a fire that can start and that's been an issue. So there's been preemptive outages forced upon the populace by the electric utility to save them from worse situations than simply not having power. So you're facing that. And I'm sure not just in California, just they’re in the news all the time about it.

Baldwin: I saw someone that survived the wildfire last week out here, which got closer to home than I really wanted or I ever wanted to imagine. The one thing that I started to recognize about it was the amount of time to discover the location of where we cause this fire. There are so many enterprises, and fault location is gonna be something that needs to be prioritized in our industry. The faster we can locate where this wire whacked into a tree just in general is so important because we have only a limited amount of resource in a gigantic territory out here. So we can't be everywhere. And it is that, how do you model that correctly? What new technologies come in that we can chase that down? I think it's going to definitely help.

Paul: Certainly there's a lot of people working on it. We do. We've got techniques not all of which have been commercialized yet for fault location, but if you have a good model and you have good measurements being taken by what they call digital fault recorders or similar equipment, digital monitoring equipment, we know what those values are at various locations. We can tell you where the fault is.

The equipment itself, relays that are being made, are small computers these days—have been for the last 40 years, 30 years—and have the ability to find or show a fault location as part of their output when a fault does occur. So in other words, a lot is being done in that area, but we're not there yet. And maybe we'll never be perfect, but there is a lot of work going on, on the fault location topic.

Closing thoughts

Katie: Well, I found this conversation fascinating and just to learn more about CAPE software and just the wide-ranging ramifications. So interesting. Do you have any closing thoughts?

Paul: There are a lot of things that I think about, but I would say that mankind has always had the talent when a problem is identified, had the talent come to the surface somehow to solve a problem. Now, do we solve it fast enough? No engineer is ever satisfied with that response because the answer would be, no, you don't solve it fast enough, but we do get there somehow. And so I think while there are definitely challenges, constantly awaiting us that we also have the talent of our people, and that's people from all over. You heard me say we had people from India, from China, from Mexico and from the US and from Europe on our team, because we only hired what we felt was the best people for the job.

They're out there and they will continue to be out there in various ways to solve problems. So I guess I'd like to remain optimistic and we must be optimistic. That's my contribution from the engineering point of view.

Katie: Thank you. I like that, that's very well rounded. Baldwin?

Baldwin: The one thing that working with Paul and I've been doing this for quite a while, and this is my passion, or are these system models and getting them right is that no matter what you call this data analytics, data analysis, asset management, it comes down to the same couple of things as in, we need people to communicate. Different departments, as the workforce labor is getting smaller to Paul's point. We need people from all angles to help us solve this problem.

But specifically, if you have somebody in compliance, we need to leverage compliance. For example, PRC 27, a wider area study coordination to make sure the grid is accurate. We need to be able to leverage those kinds of initiatives to solve all their problems because we'll discover, Hey, working on this problem, this other department may have a better solution or they needed help. And as we're pulling it together, the elegant solution usually presents itself.

And my message for younger engineers is you really gotta get out there and see what it's so easy to be pigeonholed these days or almost be too broad, and don't be scared of someone telling you this is how we've always done it. I almost find that as a challenge. There are so many new techniques and other industries and with technology and don't be afraid to CAPE. CAPE has so many great applications that people don't even recognize that you could use to help you solve new problems.

Paul: Baldwin, I wish I had thought of your comment too, though. Communication, that amongst the different parties within the utility, between utilities and beyond how important communication is and sometimes we seem not to do it.

Baldwin: Even within our own industry, when I speak to people at ICF—for example, Heidi Larson—when she does these independent engineering evaluations, I get these moments. I'm like, Oh, that's why you're asking me this question. Or when we're collaborating, she's starting to understand my pinch points. And while we're working through them, they turn out most of the time to be things all achievable, but it's the lack of communication or even education I would say, especially on my end with renewables.

There's just so much in the market changes so quickly, as opposed to when you're working for electric, utility change tends to be a little bit slower. So we're starting to see these two things trying to slam into each other, but I can tell them the last couple of years, it's gotten a lot better engineered, especially when we get this younger labor force who's willing to go try new things. And quite honestly, the internet and technology has just made the world a smaller place.

Katie: To both of you. Thank you so much for being here today. I'm happy we're able to do this.

Baldwin: Thank you for having us.

Paul: Welcome.