>> Tom Williams: Hi everybody.
How is the volume?
Good.
Thank you, Irfan, and thanks to your team.
So we did not submit our slides before this workshop.
I think Irfan had no idea what we were going to be talking
about.
The synergy of the topics today is astounding to me how one deck leads to another
deck and all these topics are kind of interweaved you might say.
So if that doesn't define synergy I don't know what does.
And I think it's also evidence that we've picked a really good topic and some really
good problems to solve.
I'm going to start with a story and it's somewhat close to home.
And I think it's a way to dramatize for you the idea of, this idea that
the adversary is already in your network.
Forget about getting breached.
It's already there.
So I spend my time between Reno and Sacramento.
I work in Folsom in the Sacramento area and more about that in a little bit.
I have a house in Reno also.
So let's talk about Las Vegas.
Let's talk about Stephen Paddock.
Have I got your attention now?
This guy had a house just down the street from
me in Reno.
It turns out he had been stockpiling guns there for two years as well as his
place in Mesquite and who knows what other places.
You got the cyber context.
Right?
The adversary is already in your network.
You're working with the network every day going about
your business but they're in there exfiltrating data.
They're in there stealing, whatever they're doing, they're already in
there.
You should assume that they're already in your network.
Approach cyber security from that point of view.
I thought it would be good to call out some of the main learnings I've had today so far
just as a thank you to everybody and as a way to try to tie all of these topics together.
So Irfan, the idea of systemic security.
If you take one thing from this workshop, ponder
that.
It's a super powerful concept.
Rajit's question, what are we securing?
Isn't that an interesting question?
Think about the CIA, this traditional IT cyber security point of view
of confidentiality, integrity and availability.
I'll start this stuff in a minute.
I'm just ad hoc here.
We have to flip that exactly around for what we're doing.
Availability first which is reliability and as I'll make the case shortly
it's also protection and safety.
Then it's integrity.
That's what you really care about, the integrity of the data.
You don't want that SCADA signal to change.
That SCADA signal has to be, has to arrive just as it was sent
without interruption.
And by the way, there's confidentiality.
When do we really care about confidentiality?
When we're dealing with meters.
Meters are the cash register for the grid.
We care about confidentiality there.
Take that CIA and flip it around.
That's the kind of cyber security we want.
Doug Campbell's idea that we need to assume that our connections will be to the public
internet.
This idea – so the semiprivate and private networks.
First of all, not all of us own our own fiber.
Right?
And the carriers are already communicating the
telecommunications carriers that they're getting out of copper.
What does that mean?
No T1s.
MPLS will change as we know it.
We should assume that you're going to get an
ethernet handoff.
You're going to connect to someone's Cloud and that's what you need
to secure.
So thank you, Doug, for bringing that up.
I'm super interested in the research that Jay and Dinesh presented on the protocols
and I think here are two guys that are really modeling
the IT OT convergence that I will talk about a little more.
Ok.
One more story and then we're going to get started.
After this slide – this is the nonattribution slides.
If you attribute anything I say to me or the ISO I
will disown ever having met you or being here.
That's what this slide says.
Ok?
Nonattribution, an important, a very important security concept, nonattribution.
So it's rude to talk about acronyms without expanding them.
I'm going to give you lots of acronyms and while I'm speaking I'm not going
to expand them but at least I'm being a little polite and you can stare at this while
I'm telling a story.
So there's this myth that the inventors of our key protocols such as TCP,
so Vinton Cerf, now a vice president at Google.
We hear all the time that no one thought about security back then.
Why didn't they think about security?
Obviously, we needed security.
They did think about security.
Vinton Cerf pushed really hard to build encryption into TCP to make it
nonoptional.
What was his use case at the time?
Was there any commercial context for the internet at the time?
Nada.
This was a military context.
He was building, he wanted to build encryption into the TCP protocol.
Who blocked it?
NSA.
Think about that.
So we'll talk about this idea of ubiquitous surveillance
a little bit.
Let's get started.
I'm going to give you some basics for cyber security.
But here's the deal.
I'm an IT guy.
I'm a cyber guy.
I can talk that stuff all day long and it's no
challenge for me.
I wanted to challenge myself.
So I want to model for you IT OT convergence.
I know all this IT stuff.
I know cyber really well.
Until a couple years ago, I hardly knew anything about OT.
And that became for me in my work at the California Independent System Operator.
And if you don't know what an ISO is, hang in there.
That became a career liability for me.
I could not advance unless I started to hang out
with power engineers, started to hang out with the OT guys.
So I'm going to talk more about OT today and if you want to – I encourage
you to educate me if I screw up.
Ok?
Do we have a deal?
If I start stepping on myself help me out, help me learn.
If we don't keep learning, we're done.
Notwithstanding what I just said, it is imperative that you have a strong information
security management program in whatever form your enterprise takes.
You've got to have the information security management system.
You should have an enterprise information security architecture.
This is a document that describes the enterprise approach to security architecture.
This is a reference document.
You've got to have a strong compliance program and that program
needs to be focused on operational situational awareness, on grid awareness.
You need to measure everything you're doing and you need a framework to keep yourself
honest.
When you see the little infinity symbol today I kind of made this up.
This is – well, not exactly.
So this is a symbol for continuum also.
I think this is maybe the most important concept I want to leave with you today, this
idea of continuum.
And we'll talk more about that in a little bit.
And you've got to hire well and you've got to remain calm.
So we had some excitement.
I was unable to connect my Chromebook to the larger display.
So just remain calm.
Robert Katz jumped in.
This is his laptop, thank you.
And we're good to go.
There is generally an answer if you remain calm.
Why does cyber security fail?
There are – there are some really good reasons it fails.
It's hard.
It's expensive.
It's prone to obsolescence.
It will fail if it's coupled with the application layer.
We've touched on that theme several times today.
We're going to talk about the importance of putting security at
the network layer.
I love this next one.
Cyber security is a fool's errand.
What does that mean?
Are we fools?
Well, no.
But it's a challenge.
Why is it a challenge?
The problem is asymmetric.
What does that mean?
By way of example of an asymmetric problem, if you have a vulnerability in your
system and you patch it, what have you done in addition to fixing that vulnerability?
You've change the hash of that OS.
You've introduced more vulnerabilities.
Every time you patch, you fix and you add.
How can you keep up with that?
We need to take a different look at this idea of vulnerabilities which
are very difficult to assign value to.
In my opinion, the things on the internet, and DERs are things, should not communicate
directly.
Their communication should be brokered through a trusted Cloud.
We'll talk more about that in a little bit.
And we've touched on several times this last idea.
When we think that encryption and authentication
are enough, we're done.
We've lost the battle.
You've got to look beyond encryption and beyond authentication.
I didn't say ignore them.
I said look beyond them.
[Inaudible audience] I would prefer to ramble.
Are you good with that?
Ok.
But if you could make a note of – [Inaudible Audience]
We'll come to that.
We'll come to that.
Big topic.
Great topic.
I'm going to go very quickly over this slide here because I've got more
important slides.
But you should know what security architecture is.
This is what I do.
I align the security to the business mission.
I am in service to the business mission.
I am not dictating solutions.
I am figuring out what the business does an applying adequate
security to it.
This is what a security architect does.
This is what a security architecture program does.
We document strategic technology adoption.
We assess the risk of disruptive changes to the technologies, to the protocols.
And we enforce the point that everything we're doing
in cyber security for distributed energy resources begins and ends with the network.
Stay out of the application, get it right in the
network.
Here's the continuum concept.
What I want to drive home here is that we are moving
from one type of grid to another type of grid.
But it's not happening very quickly.
Probably won't happen very quickly.
But in retrospect it happened really fast.
So that's a paradox.
Look where we are with solar today.
We have at any one point on the California ISO grid 25 percent of that supply coming
from solar.
Who would have thought we would be in that place today?
So it feels slow sometimes but it's actually moving really
fast.
That's an interesting paradox.
So we're becoming more distributed, less centralized, more digital, less analog, more precise, less
forgiving.
This one is interesting.
At the ISO we do four second telemetry to the field.
We have about 1,000 nodes.
Every four seconds we're pulling these nodes.
Is four seconds frequent enough?
We're not sure.
Is one second better for certain use cases?
We're trying to figure that out.
Jay and I were having an interesting discussion about this.
I see a future with more automation and less supervision.
And let me jump right to the next one, more machine learning, less human learning.
I didn't say less human involvement.
Right?
It's less human learning.
And what's key to machine learning?
Unsupervised machine learning.
It can't be a human saying "Ah, an anomaly.
I'm going to go change my statistical model and we'll
run that again."
That will never work.
There's too much data.
The machine has to do that.
That's what unsupervised machine learning is.
And complex event processing means this ability for the machine to ingest massive
amounts of data beyond human comprehension and produce a result.
The percentage of meaningful log entries is just vanishingly small.
Something like 4 in 10,000 log entries in the latest report I've seen from Splunk, 4
in 10,000 are actually actionable.
Can a human do that?
I would say probably not.
And we're going to see less emphasis in my opinion
on forensic analysis because we don't have time for it.
We're doing this stuff in real time.
Here's the convergence piece.
So we have a digital substation.
We have really important issues about volt and VAR optimization and
these issues are accelerating that OT IT convergence.
I've talk to several utilities now who are putting OT and IT under one
executive, even on the same team.
This is an awesome trend in my opinion.
And I feel very strongly about the next one, OT has to
lead this.
IT learns and supports.
Find the people in IT who want to learn OT and make
them hang around OT and magic will start to happen.
That's what I believe.
And we can train the OT and IT.
OT is difficult.
IT is nothing for OT to learn compared to what they've
already learned.
This is the slide with the answers.
In my opinion, this is not the all-inclusive list but this
is a list of top things, top cyber security principles, goals, requirements that you've
got to pay attention to specific to distributed energy
resources.
You need authentication before access.
That's called pre-authentication for those things, those nodes, those DERs.
You need to white list those talkers in the network.
You need to white list those protocols.
That system that you've developed has to be immune to fuzzing.
If you throw garbage at it in the form or parameters or in some other form, that system
has to survive that.
It can't crash.
There's no time for it to crash because if it crashes your
advanced volt VAR optimization algorithms will fail because those packets drop, because
you've got no data.
You've got to have the data.
You've got to have the resilience.
It must have immunity to service denial, distributed
denial of service attacks.
It has to sniff, sense, smell those attacks and go in another
direction.
There cannot be a single point of failure in the network.
You need encrypted, diverse software defined networks and software defined wide area
networks and nonrepudiation of message delivery and receipt.
Nonrepudiation basically means there can be no digital doubt, digital
doubt that a message was sent or a message was received.
You cannot falsely repudiate, deny having sent a message or having
received a message.
That's what nonrepudiation is about.
And there from a textbook point of view, there's been only one way to
do that and that's asymmetric encryption and PKI.
It's not the only way to do that.
What's an ISO?
That's an ISO.
That's the ISO in Folsom, California.
Folsom is about 25 miles east of Sacramento.
That's a picture of campus, new from about five or six years
ago.
You can see in the bottom right corner a bunch of solar covered parking.
There's a bunch of solar in the back.
You can see a water chiller in the back.
We do lots of visits and we welcome visitors.
And that's a picture of our advanced control center.
We have another data center in Lincoln, California
about 30 miles northwest of Folsom.
And we are one of nine ISOs in North America.
More about that in a little bit.
But we are nonprofit.
We are a public benefits corporation.
Our mission is to manage the flow of electricity.
And to some extent we do that through markets.
I should say to an increasing extent we do that through markets.
Increasing means the old way of bilateral contracts and the new way of markets.
More and more markets are driving reliability which I think is a counterintuitive concept
that markets could drive reliability.
These are the nine ISOs in North America.
FERC order 888 from over 20 years ago basically asserts the advantages of competition in wholesale
electricity markets through the creation of an independent body.
That's as far as the order went.
And this idea of an ISO grew out of that and an RTO, Regional Transmission
Organization.
RTO is basically multistate versus single state.
One unique characteristics of the ISOs we're the only ISO formed by legislative statute.
So our board, our governance, our board, all of that is tied to legislation which is raising
some interesting complications for us right now.
Most of the other ISOs grew out of power pools from many decades ago.
And we have a responsibility for maintaining automatic generation control on the grid.
That means keeping that frequency at 60 hertz.
That means safety.
That means reliability.
That's what we need to support with cybersecurity.
Now we faced a really interesting security challenge at the ISO that had to do with
divestiture of generation assets owned by the investor owned utilities which is part
of the deregulation.
For example, PG&E sold three big generators to Duke Energy in 1997.
And you can see kind of the news article in a
very faded background there.
So we went from a very simple model on the left where the utility
through remote terminal units connects to generators.
Easy, no problem.
A somewhat more complicated unit model where the ISO
using ICCP protocol would connect to the utility and get data indirectly from those
RTUs.
Ok.
We can manage that.
But now the utility sold the generators to a merchant utility.
Now what?
Oh, we'll just connect – the utility will continue to give
us that data through ICCP.
What do you think the answer was to that?
Hell no was the answer.
Figure it out yourself.
This is what really happened many years ago.
So we had to invent a secure telemetry method for connecting
directly to RTUs or we have the concept of a RIG, a Remote Intelligence Gateway.
So we pioneered PKI.
This is going back almost 20 years when hardly anyone knew what public
key infrastructure was.
We put a digital certificate on a field device.
What could go wrong with that?
And then secure that telemetry over SSL at the time and now we call it
TLS.
And it's worked beautifully for many years.
But it is a constant maintenance headache.
The certificates are expiring.
We have people in the field who understandably don't know
anything about digital certificates or certificate authorities or PKI.
Not that they're not willing to learn.
They learn faster than most.
But it's not their job and it's not their priority.
And it's from one point of view a troublesome decision for security to embed
that kind of responsibility in the field.
And by the way, the ISO doesn't own the field device.
So we have to work with the owner of the site to apply appropriate security.
So this is something a little bit different than
the ISO.
There are 38 balancing authorities.
A balancing authority is managing that equilibrium of
supply and demand in real time across a larger, across what in the area known as a
balancing authority.
There are 38 of those in the WECC region.
We'll talk more about that number in a little bit.
The ISO now has a western energy imbalance market.
In eight states in the west, every five minutes we
clear a price for energy that buyers and sellers agree on across eight states in the west.
So at a transmission level, we're still California.
At a market level, we're eight states.
And that's the energy imbalance market.
Every five minutes, 5,000 nodes crunch and we get a market price, a clearing price that
all consumers in that market benefit from.
And there have been major benefits to many consumers because of the energy imbalance
market.
One reason we're interested in this from an ISO point of view is we have more
solar than we can use.
And we'll talk about that in a little bit.
What if we could export some of that solar?
What if the west could take advantage of that solar?
That's one of the things the energy imbalance market is doing.
This is an interesting graphic.
It's showing conceptually on the right side the 38
balancing authorities in the west.
On the left side, it's showing this concept of a reserve
sharing group.
So when something goes really wrong in the grid, when there's a
frequency disturbance, in the old model there was this idea of a resource sharing group
and the resolution to the model was pretty manual.
Phone calls basically, humans jumping on the phone, getting involved and
getting power from somewhere else or keeping the grid flowing in a safe way.
What we're discovering at the ISO is that increasingly we can do this in an automated
way through market dispatches.
So the software is dispatching energy in an optimized
way to solve frequency disturbances.
This is a really interesting idea and we're very
excited about it.
This is a random slide but interesting.
A group of us recently returned from Santiago, Chile.
Chilean ISO, who knew?
Right?
So really interesting stuff going on in Chile.
They're doing a lot of solar.
They have a very unstable grid right now.
They have four main regions that they're trying to knit together.
They're implementing frequency control from a centralized systemic point of view
for the first time.
And we had the tremendous opportunity to advise them on our California
experience which seems to mirror their experience in Chile very, very closely.
So it's very exciting to see how we could be of
service to another entity and in Chile of all places.
So really interesting things going on there.
Got some first principles here.
This is maybe the most important slide.
Forget about confidentiality.
Forget about integrity.
Think about availability and safety and protection.
You have to isolate that fault.
Right?
That's number one.
You've got the safety of humans and you've got expensive gear out there.
If there's a fault, you've got to isolate it.
You've got to contain it.
And by the way, isn't that a lot like cyber security?
If there is a breach you better contain it.
So it's very interesting how these two worlds mirror each
other.
I want to challenge us to take another look at the NERC definition of a protection system
to see if it's relevant to distributed energy resources.
These are taken, these comments are taken straight from the NERC glossary of terms
which is easily Googleable, Googleable.
And I challenge you.
We'll share these slides.
I challenge you to think whether this remains relevant for DER.
And I've had some very interesting conversations today about
whether NERC is moving fast enough for us.
We've got some supporting principles.
We've covered these all day long.
It doesn't hurt to repeat them.
Reliability, resilience, sustainability, efficiency and cyber security.
The order is important there I would say.
This next one is close to home for me.
What we come up with here has to grow out of our practical
experience and we have to support that practical experience with academic research.
I feel very strongly that we cannot develop these standards in a vacuum.
We have to come from a practical point of view of
what works.
And you have to talk to the people who are touching these systems to find
out what works.
And so often, IT is not talking to the right people.
So this is the IT and OT convergence.
And we've talked about this idea of standardizing and certifying best practices.
We're going to talk about the interface between
transmission and distribution a little more in a
minute.
We have a definition of DER at the ISO.
So these are all the distributed energy resources, all the energy resources connected
at the distribution level on either side of the
customer meter plus the supervisory control, plus the telecommunications.
That's how we are approaching this problem at the ISO.
How do we maintain voltage control?
What happens when you have a DER depending upon where you put it, you might introduce
load on the system.
Right?
Does your substation know about that?
We have to figure these things out.
You have to figure out how the DER can be a reliable add on to this
larger system.
Where is your attack surface now and how do we manage that over the wide
area network?
Attack surface, the best way to depict attack surface is what's this?
It's a big attack surface.
What's this, right?
It's a little attack surface.
At the ISO we're moving toward no DMZ.
Have you ever heard anyone say that?
We're eliminating our DMZ.
How are we doing that?
We're putting an application delivery controller on the edge.
And that controller decides what's allowed within the network.
And then you don't have these VLANs and subnets hanging off your edge firewall.
Eliminate your DMZ.
This concept of a high DER future, a future with a high level of
distributed energy resources.
So the whole point of why we're meeting is we need to do
this in a way without compromising the reliability, the availability, the safety and the
protection.
In California we are driven by RPS mandates, Renewable Portfolio Standard.
We have a very aggressive onboarding of distributed energy resources at a legislative
level in California.
Customers care about the economic advantages of DERs.
Of course, we have to pay attention to that.
Microgrids should buffer the macro grid, the mega grid from the
variability that is inherent in these variable resources.
Perhaps microgrids can offer additional protection from a critical infrastructure
point of view.
An ISO has to care about visibility.
If you've got enough DER – we don't know what enough means but if
you've got enough DER on the other side of the meter, on the retail side, on the customer
side of the meter and the ISO doesn't have the ability to interrogate that, what does
that mean?
It means the ISO is losing operational visibility and maybe that's important.
What's that?
Thank you.
That's a duck curve.
So you may not know that Clyde Louden one of the principal engineers at the ISO.
Anyone know Clyde?
He came up with this concept of the duck curve.
Wikipedia has a surprisingly good article about the duck
curve.
You learned about the duck curve today.
The idea is that you have this concept of net load or net demand.
And net load or net demand, this is the amount of demand you
have to satisfy with non-variable resources.
So hold that thought.
And you can see that as the sun comes up and the solar panels are providing more and
more supply, the net load goes down because we don't need those traditional generators
as much.
And that's the belly of the duck.
And the inverse of that is at the other end of
the afternoon especially when most people are going home from work and turning on
their lights and we're losing the sun.
And now we have to ramp up those traditional resources and that's the neck of the duck.
This is the same thing – you wonder ok, where did we get – who did that?
Where's the data?
There's the data.
That's our PI system.
You know what PI is?
Plant infrastructure.
This is OSI soft.
This is a graph just any old day from one of our – this is one of our PI
displays.
And at the bottom there, the belly of the duck, that's the net load.
So we've integrated that for our operational visibility.
This is what the operators on the floor are looking at.
And that's clearly a duck.
What's that?
That's the August 21st eclipse.
California was in the 55 to 80 percent occultation path from north to south.
Even though we weren't in the path of totality we
had a significant impact on the grid.
What happened during the time when normally the
ducks belly would be forming?
The sun was covered partially, less solar supply.
That's the yellow line there.
So we worked.
We did a lot of studies and we saw that over roughly
three to three-and-a-half-hour period our solar output would be dropping by 55 percent
initially over mostly utility scale solar, about 1,900 PV utility scale plants.
And on the other end, it would be ramping up.
And the ramp was very impressive, the ramp up.
It was about 60 to 70 megawatts per minute down,
150 megawatts per minute at the peak on the other side, five times what we normally
see in a day.
How did we manage that?
Through the energy imbalance market.
That was the main factor.
Through calls on conservation.
We have a program called flex alerts.
And by calling on more traditional resources.
So this was an extremely well-choreographed event
and a success for us.
But forget that.
What does it really show?
It really shows the ability of software to handle huge variability in
generation.
So that's a really interesting point to drive home.
This is simply depicting the duck on eclipse day.
See that highlighted part in the middle, what would normally be the belly, losing the
sun.
The belly goes up and getting the sun back.
So during this middle phase here we had to supply from traditional resources or we
were calling on conservation or we were using markets to optimize the grid.
On the left there is we have, we have an app – the ISO
has an app for iPhone and Android and there's a lot of data on that app and it's
really quite fun to use.
ESDER, Energy Storage and Distributed Energy Resources, I just wanted to make you
aware very briefly of this initiative.
We are now in phase three.
So we are approaching from many points of view the complexities
of integrated energy storage and distributed energy resources with the grid.
We have particular focus on telecommunications and
telemetry.
Thank you, Irfan.
And we are providing new solutions to our customers
specifically for telecommunications and telemetry.
We want these communications to be network agnostic, equivalent security on public
networks and private networks, protocol agnostic.
They should handle any underlying business protocol.
We don't want customers to have to maintain digital certificates.
We want rapid onboarding.
And I'll leave it at that in the interest of time.
So what we've come up with at the ISO to provide a solution following an extended
stakeholder involvement process was an implementation of software defined wide area
networks specific to DNP3 telemetry.
Last year some of you were here.
We presented this from a conceptual point of view.
We're live now with our first handful of customers and we're using dispersive technologies for
this purpose.
I love this diagram of SD- WAN.
This is, it quickly conveys this idea of process connected Clouds, multiple Clouds
connected at the process level with those familiar Visio flowchart diagrams.
These are all the attributes, the strong security and availability
attributes of the software defined wide area network.
And I would specifically emphasize the ability, the capability of a software defined
network to protect against distributed denial of service attacks by sniffing congestion
and moving around it, simply moving around it.
Why?
You can do it in software now like that.
You don't have to log into a router anymore.
You've separated your control plane from your data plane and now you can rapidly
make decisions using software in a control Cloud.
That's the beauty of SDN.
These are the three telemetry options for DNP3 that the
ISO offers today.
There's traditional T1 over MPLS.
There's an IPSEC option in the middle and the new dispersive option is shown
on the left.
So this is a high level conceptual diagram of the three ways in which
customers connect to us and we're changing this, optimizing this, improving
this as we go.
This is the dispersive solution, information easily available on CAISO.com.
Just Google CASIO, CISDN.
That's the name of the dispersive product.
And to conclude, some discussion about the future grid and the future grid architecture.
It's both decentralized and integrated.
It's integrated with layers and these layers – think
of it as an onion.
These are concentric self-optimized layers in a network and they're
optimizing buildings, microgrids.
And this concept of a distribution system operator
could be extremely disruptive for us.
And in fact, it could be disruptive for an ISO.
So I'm up here telling you that one of the answers
is a distribution system operator in addition to an independent system operator.
What does that mean?
We need to talk about it.
And maybe the idea of an ISO needs to change.
And all the context we need to keep in mind, the local economies, closing the loop on
waste, supporting the renewables portfolio standard and reliably, reliably, reliably.
So this is a little bit about – we'll share
these slides.
This is the concept of a distribution system operator which is an ISO at the distribution
level.
That's the main concept.
And finally, to hammer home this concept of integrated
and decentralized management of the larger grid which is now both the wholesale
and the retail side, both the transmission and
the distribution side.
And what does that interface look like between the transmission and
the distribution and how do we get that right, safely, reliably, securely?
And this is just for fun.
Do you know true to size?
This is an awesome website.
So this is the Mercader projection.
To take a sphere and flatten it, right?
You distort the things at the polls.
Is Greenland really that big?
Of course not.
This is Greenland down here.
This is Alaska down here.
So keep your DERspective is what I wanted to leave with you
today.
Thank you very much.
I've really enjoyed talking to you.
Không có nhận xét nào:
Đăng nhận xét