In the cloud networking course, we will see what the network needs to do to enable cloud computing. We will explore current practice by talking to leading industry experts, as well as looking into interesting new research that might shape the cloud network’s future. This course will allow us to explore in-depth the challenges for cloud networking—how do we build a network infrastructure that provides the agility to deploy virtual networks on a shared infrastructure, that enables both efficient transfer of big data and low latency communication, and that enables applications to be federated across countries and continents? Examining how these objectives are met will set the stage for the rest of the course. This course places an emphasis on both operations and design rationale—i.e., how things work and why they were designed this way. We're excited to start the course with you and take a look inside what has become the critical communications infrastructure for many applications today.
Interview - Bruce Maggs (2015)
Loading...
Del curso dictado por University of Illinois at Urbana-Champaign
Cloud Networking
180 calificaciones
University of Illinois at Urbana-Champaign
180 calificaciones
Curso 5 de 6 en SpecializationCloud Computing
De la lección
Week 4
Zooming out from within data centers, this week will examine other pieces of the cloud networking ecosystem: inter-data center WAN connectivity, content distribution networks, end-user Internet connectivity, and application interactions with the network. We’ll learn what novel techniques in each of these areas are pushing the cloud’s capabilities farther.
Conoce a los instructores
P. Brighten GodfreyAssociate Professor
Department of Computer Science
Ankit SinglaAssistant Professor
Department of Computer Science, ETH Zürich
[MUSIC]
Hi, today we are happy to be sitting down with Bruce Maggs.
Bruce is the Pelham Wilder chair computer science at Duke University.
And he's also Vice President for Research at Akamai.
>> Thanks guys. >> So what kind of
content is Akamai serving?
Who's using Akamai?
>> Well, there's a very diverse set of customers who use Akamai.
There are tens of thousands of different
websites that use Akamai to assist in delivering their content.
There are a few major categories so one category you've probably all heard about
is video, so the largest source of bytes delivered is video.
So movies would be a good example of that,
we have some customers that use Akamai to deliver movies.
There's also live move events where there's action that's being
captured in real time and immediately sent out over the internet.
And occasionally those events can be very, very large.
Another big category of byte served is software downloads.
Software downloads are sometimes served when a user is explicitly
clicking on a button on a web page to download something.
But, oftentimes they're taking place in the background, and
it's just some kind of software update program that goes and makes requests to
Akamai to fetch the latest version without the user necessarily being involved.
And these two categories of traffic are what we would call
latency insensitive categories where there may be bulk data, but,
there's some tolerance for a little delay at the beginning, whether it's a delay in
a few seconds of buffering before the movie starts or
maybe if a software download's going on in the background, you know,
the user is not going to know whether it started one second earlier or later.
There's an entirely different category, which is the latency sensitive traffic.
And the best, maybe the most important example here is e-commerce websites.
Websites where the content provider is offering a product, offering
a service selling something online, and they want to have snappy performance.
And, there have been a number of studies that show that, even small reductions in
latency in the time it takes to get a response when you interact with a website,
can greatly improve the experience for end users and
measurably increase say stays.
So that's one of the reasons why we fought so
hard to get our servers deployed as close to the end users as possible.
Although I will say that another reason for
having this very widespread deployment is that for
certain categories of traffic like video, or software downloads where you have
to serve a lot of bytes, you also need to find a lot of bandwidth on the Internet.
And that's another reason for a broad employment.
I'm going to just give you one more category,
although I could probably go on and on, but, increasingly websites
who are subject to periodic attacks of various
sorts, use the CDN as a front end to provide the scale
to thwart denial of service attacks and
to implement various kinds of sanity checks on user inputs to,
to shield the content providers back-end servers from malicious traffic.
An example in this category would be things like banks and government websites.
>> Mm-hm.
>> Exactly.
>> So, you mention these two categories in particular and [INAUDIBLE] of service.
But in regards to streaming movies and software downloads being one category,
latency sensitive traffic, like serving websites being another category.
How do the needs of these two kinds of applications differ?
What do they want from Akamai, do different things for them?
What changes?
>> Well, I think if you look at the primary needs in
the case of the latency sensitive traffic,
It's, it's important that the user gets a response as quickly as possibly,
which means, if we can, we will want to cache a copy of whatever object
the end-users are going to require as close to the end-user as possible.
>> So you care about seconds, hundreds of milliseconds, tens of milliseconds?
Seconds? Individual milliseconds?
>> Probably not individual milliseconds but,
certainly hundreds of milliseconds and maybe tens of milliseconds.
You want to see.
Yeah, sort of along that scale.
You want to shave 100 milliseconds off the page or
maybe off each object off the page.
That kinds of scale.
The video traffic is a little different.
There, you really want to provide
a once you sort of determine the capacity of the channel.
Between the end user and the internet.
You just want to make sure you can sustain standing delivery of video at
whatever was the suitable encoding bitrate for that video.
>> Mm-hm.
>> Now, one thing they hadn't mentioned was that
There are also tricks that can be used to reduce latency,
even in the case where the object to be served isn't captioned.
I know that sounds contradictory or
perhaps hard to see how a CDN could do that but
if you like I could also talk a little bit about over [INAUDIBLE] absolutely.
Yes.
So there are multiple techniques I presume that go into making things work for
objects that are not cacheable.
We can discuss all of them, as many of them as you can tell us about.
Overlay routing or perhaps connection proxying for TCP.
An event.
>> Yeah, so those two are, they're very closely related.
And I don't know if the class knows a lot about the details
of the TCP protocol, so I'm going to review something very quickly.
And that is that when a client wants to make a connection,
to a server, say, a web browser connecting to a web server.
The client initiates a request to open up a TCP
connection between the client and the server.
And at the beginning, there's something called the TCP three way handshake
where the client sends a packet, we call it sin packet, the server sends a sin act.
And then the client sends back an act, and
at that point, both parties can start sending data.
But also, TCP has built into it sort of a period of time where the two try to figure
out how fast can they send data without actually causing congestion.
And it takes a little while for
them to ramp up their speed to that sort of ideal rate.
>> That's right.
>> And all of this adds latency.
And so one question is,
is there any way to speed up the initial part of a TCP connection?
One thing you could do is move the server closer to the browser.
And the reason that helps is that the reason this initial
phase in TCP is slow is that there are many round trips going back and forth.
If you reduce the latency, those round trips happen more quickly.
So one technique that employs and provides as a service for
many content providers is that we act as their front end.
So, even for content that's not cacheable.
The end user's browser will contact, hopefully, a nearby Akamai server will
Launch a new TCP connection with that server over a low
latency interaction.
Meanwhile Akamai, the Akamai server will have already opened up and
maintained for some time.
A persistent TCP connection back to the content provider.
Which might be a much greater distance and much higher latency,
but the initial TCP face has already been completed and so
the end user doesn't see any penalty for that.
So that's I guess what a kid called TTP proxy.
Where we have the ability to sort of serve as
a benevolent man in the middle, who helps terminate
the TCP connection more quickly and then passes everything through.
>> So each Akamai user facing server will then have a persistent connection
to all of the content providers or those that are sort of cached in some way.
>> Well not all content providers use this technique and
it may be that unless there have been a connection from other
users through that server to a contact provider before.
It may not have set up the connection yet.
And, furthermore, within any one data center, Akamai kind of partitions
which servers are acting on behalf of which contact providers.
So, it wouldn't really be an all-to-all work,
every Akamai server's connected to every contact provider.
That wouldn't be efficient.
>> Three server will at a particular site.
One server will kind of specialize in a certain subset of the content,
and another server might specialize in another subset, and probably these
will be overlapping in some way >> That's possible,
but I think a better way to think of it is, let's say we have a cluster with 64
servers somewhere.
So, we use a technique called consistent hashing to map
the different content providers content onto the different servers.
Any one server may be serving content for,
you know potentially hundreds or thousands of different content providers.
>> Right. >> But, if you look at any one content
provider within a cluster they might be one or two or
four depending on how popular that content provider's content is.
One or two or four servers that are helping serve the content for
that content.
Now it is also possible that we might take a content provider's content,
different parts of the page and split it across different servers.
But actually we typically, unless there's a really good reason to do that,
probably want an individual content provider to be served and
have one server, the whole page.
Or have several servers that are each capable of serving the whole page and
all the objects, rather than splitting it across servers.
>> Yeah, that makes sense.
>> Now, Ankit you asked about TCP proxy.
You also mentioned something called overlay.
And, what I've just described TCP proxying is you can kind of view that as an overlay
network in the sense that instead of having the browser
connect directly to the content provider and
using whatever path is selected by the internet routing protocols.
We've introduced an intermediate point which
is the CDM's front-end server that's now proxying the TCP connection.
So, that's a A kind of simple example of an overlay.
There's another example that Akamai uses.
Between our server, our cache,
which I've called the front end server, and the content provider's backing.
So, I said that we maintain a persistent TCP connection back to
the content provider.
In many cases that's true, but we also we may try to find a better connection than
the one that is the default path provided by the internet routing protocols.
We will also look for one hop paths that go through
another Akamai server located somewhere else.
Which is then, let's say proxying the TCP connection.
It's actually proxying an HTTP connection
from our server back to the content provider.
And we'll explore both the direct path as well as several
one hop intermediate paths and look for the best one.
Best meaning lowest latency, or lowest packet lost,
or best bandwidth, depending on the application.
>> Right, and by one hop you mean, one hop on the overlay so the part to this
intermediary might be several network ops, but this one hop in the overlay sense.
>> Yeah, that's right.
It could be actually ten router hops.
>> Right. >> But when I say one hop is,
you go through one additional Akamai server.
You could imagine having overlay paths that go through more,
but as far as I know, we don't do that.
>> But then the object that the user wants would be downloaded from
the original content server.
So maybe an intermediate Akamai server.
That's over 1 hcp connection.
And then at intermediate server streams the same object over another HTTP
connection back to the first
Akamai server that was originally hit by the web climb, is that right?
>> Yeah. So there's sort of a series of connections
and as I mentioned earlier for the direct connection from the Akamai
server back to the content provider will have set up a persistent TCP connection.
Actually a persistent HTTP connection
will also do that between lots of pairs of Akamai servers.
So that if you decide you want to pull something over an overlay connection,
TCP is already set up and ready to go and you can just start pulling the navel.
>> So, Bruce, is there any advantage to doing this at lower layers.
For example, maybe I understood this as
the intermediate node in this overlay determine if
the front end's HTTP connection can start another HTTP connection, right?
So this is at the application layer.
So is there any advantage you see in the future for example for
doing software routing?
So you do the overlay at the routing layer, and your routing in software.
>> Well I think there might be additional performance games that you
could achieve if you had even finer grained control of the network.
But as a company Akamai has tried to
stay out of managing layers Below the transport layer.
We don't operate our own backbone.
You may have heard that companies like Goggle or Microsoft
in addition to the many other things they do, they run their own network backbones.
We don't do that.
So [INAUDIBLE] does have routers here and there, but they're mostly used just so
that we can peer with other networks, make connections to other networks
with servers that are sitting right next to the routers.
So our use of
lower level technology is, it's very limited.
And we almost never, if not possibly never,
are actually doing routing.
And so we've just avoided that because it would require a whole
new branch of expertise that we would need and so
far I don't think our services suffered from not doing it that way.
>> But, you're doing overlay routing and
that overlay routing it gives you more direct control that you
can control downloads directly in form by measurements.
>> That's right.
In some sense, we are still designing our own routing algorithms.
And we're using our own policies.
And we're incorporating things that traditional Internet routing items
don't incorporate when we make decisions.
For example, we do continuous performance measurements.
We make decisions about what overlay paths to use or
which end users to serve out which servers.
Based on those performance metrics.
But we're doing that all at the application layer.
None of that is happening down at the network layer or below.
>> Right, so with thousands or tens of thousands of
potential intermediate points, even if you're just doing one hop overlay routing.
That's a big infrastructure, it's a lot of possible paths to explore.
A lot of measurements that you could be taking about
the network performances that changes over time.
>> Well we don't really do all of them.
I think, for example, we have servers all over the world, and if,
let's say we have an Akamai server in Frankfurt,
Germany and we have a content provider in Paris, France.
And we want to know is there any path that's better
than the direct path provided by the internet.
Well, we are not going to explore a one hop
overlay path where the intermediate hop is in the United States.
We just know that in advance it doesn't make any sense.
You wouldn't want to route the traffic across an ocean and back.
We're not going to look at hops in India or hops in Japan.
So we, there's a great pulling and sort of some sanity checks that go on when you
prepare your list of potential candidates that might be used for the overlay.
And furthermore, we collect history over time so we've seen in the past
which intermediate hops set succeeded in providing better performance.
And so probably, the number eligible hops might be on on the order of a few dozen,
not thousands, like you can conceive of if every single server
location that we maintain could be a candidate.
>> So then when the request comes in, because you have this history,
do you already know when the request comes in?
You generally already know what the best path is going to be?
Or are there additional probes that happen at that time?
That's a good question.
So on the one hand, I think that after you've done this for
a while you should hesitate to ever say I already know what the best anything is.
Okay because you never really know.
And the internet changes very rapidly.
And you just hope that you make a good decision as often as possible.
But, having said that,
you know one of the things we're trying to do is reduce latency.
So the idea of waiting until you get the request, and
then doing a bunch of measurements, that won't work.
And furthermore, you because conditions can change pretty quickly on the internet,
the routing paths can change, or even more quickly the congestion may come and go.
You have to do kind of fairly continuous measurements
on a periodic scale on the order of minutes or less.
So they can look back and say hey, I just did a measurement and
this pass seemed good but this one seemed poor.
And so that's what we're doing for the overlay network.
For example, between various pairs of optimized servers or
may say server locations there's a cost continuous for
sending packets back and forth, seeing what happens.
But, you have some way of gauging whether the performance is good between that pair
or whether that's a link you might want to use as part of an overlay routing path.
>> And how big are the differences in performance if you pick sort of
the default internet path, that's what most of us are doing.
Versus a path that you've had some amount of control over
because of the overlay routing.
>> Well there's no simple answer to that.
In some cases, it's the difference between the packets not getting through at all and
the packets getting through.
And that's the case where you really want to be sure you picked the right one.
Other times it might be a difference of shaving off a little wait and
see, or getting a little more bandwidth, and it's not as pronounced.
But really what you want to avoid is
cases where you can't even provide the service using the default path.
>> And that would be because of some routing anomaly or
a spike in congestion, or- >> Yes, those would be two good examples.
>> So, the ping Akamai servers is one thing.
There's a small number of, in comparison to the end credit large, there's
a relatively small number of end points that you're considering for the overlay.
What about when you're talking about routes to IP prefixes?
Do you also do the same sort of exercise there, and if so, how do you steal that?
>> Yeah, that's a good question.
It is a much more difficult problem.
What your asking about is how do we figure out when we get a request for
an end user for content, which of our servers we should send a request to.
And again, depending of what type of content that is,
our goal would be, well we just got to.
We've just got to have the lowest latency performance.
Or we know we need to provide you know, a steady through quick for a video.
What's more complicated here as you said is there are many more
locations of users around the world that our roughly 2500 locations of servers.
And another complication is that we only control one end, the server side.
We don't control the client side.
So, we have to do something different.
We can't just do continuous probing between every client and
in the set of candidate optimize servers.
We don't do that.
Instead what we do is we try to build out a map of
the internet that captures both its topology but
also its performance in real time.
And at a very high level, it works something like this.
We collect every day, all of the requests that are made for
service to and in particular,
we collect the DNS resolution requests that are sent to us.
So, for example, we might have a customer.
We'll just call them xyz.com.
And I just made that up, so if there is a real xyz.com, it's a coincidence.
And they've decided they're going to use Akamai to deliver some of their content.
And they created a domain called ak.xyz.com,
and whenever they have a URL that uses that domain name Ak.xyz.com,
that's a URL where the request is going to come to ocomi and
we'll look at the rest of the URL and we'll hand over an object.
Well, what happens is when the browser goes to that webpage,
either goes toward the URL that stars with ak.xyx.com or
downloads a page that has some embedded objects whose URLs start with that domain.
We'll receive a request from whatever resolving name server or
local name server the browser's using will come into Akamai and
say hey, can you resolve this name ak.xyz.com?
That's where we have to make the decision about where we want to serve this user.
Now, we may not know precisely which client is making the requests.
We'll see the network address,
the IP address, of the local resolving name server.
More and more we may see something called the EDNS zero extension
which a local name server may provide to us.
Which is the slash 24 prefix that the user's in.
And based on that, we want to decide where to serve the user.
Sometimes it's really easy.
It could be a no brainer.
So, for example, if Akamai server at the University of Illinois and
you guys in your office go to a website
that's using Akamai's services, and you're using a campus name server,
we're going to get that request and say oh, we're already on the campus.
We'll serve them from the server there, okay?
And that's what we like best.
We like it to be an easy decision.
And maybe half the traffic is served from those sort of
no brainer decisions where we got a partnership with some organization.
And it's sort of, unless there's a catastrophe, we always do the same thing
because it's always the right choice.
But we can't do that everywhere, so
we also have a lot of servers in a large commercial co-location centers,
where we're just one of many tenants, and we potentially could serve
users from a lot of different places out of that co-location center.
And the question is, given that all the users and
all of these co-location centers, which ones should we serve from where?
So I'm giving you a little background, now I'm going to try to say, how we do it?
So, we collect the DNS requests that we received over the course of a day and
we sort them, we actually sort them and we extract
roughly speaking the top 300,000 name servers
that sent us requests in the previous day, once it sent the most requests.
And this'll be like 99 or maybe even more 99.5% of the requests.
I don't remember the exact numbers but this is the idea.
That we get a few hundred thousand name servers that are responsible for
sending us most of the requests.
Then what we do is, for each of those named servers, we
use a utility, maybe the class assertive called Traceroute, to map out the path
from maybe a few dozen Akamai servers to that named server.
And we look at these paths from all these different Akamai server locations,
and we find out where do they intersect?
Where's the common point so that all servers,
any one of our servers if we tried to send traffic to that
name server would have to go through this point.
We call that a core point.
We identify a few tens of thousands of core points around the internet and
we associate with each core point the name server set up behind it.
The other side of the core point from the Akamai servers.
And then throughout the day, each core point gets pinged
perhaps from a few dozen or more optimized server locations to see
what's our path like right now from these server locations to that core point?
And we assume that any requests that
we get from the name server should be optimize
according to the best path to the core point put that name server spot.
And here I'm assuming that, you know the name server is a good.
Proxy for the end users that when we get
a request from a name server, if we optimized what we think would be best for
the name server then it's also best for the end user.
>> So, you're using the name query performance optimization to decide how
to ultimately deliver the web objects then >> Yeah, yeah.
I think that is fair to say.
In the earliest stage, before the EDNS serial extension,
where the name server can now actually tell us something about the client.
>> Right. >> We used to only have the nameserver's
IP address.
And so the only thing we could do is you know,
just hope that the names server was a good representative,
the path to the name server was a good representative path for
the path to the actual clients that were using the names.
>> Well is EDNS widely deployed now?
I mean, it seems nice it gives you more, you don't have to
use this proxy, that's an overloaded term here in this conversation, but proxy for
the actual client.
The name server, that's what you had to use, but
now you can see the actual client prefix, IP prefix.
If the NSC was there.
So, is it widely deployed now?
>> I think it's widely deployed in the sense that there
are certain very heavily used name services like
Google's name service, then it does deploy it, and
we do accept the EDNS0 prefix from Google, and we act on it.
In terms of percentage of local name servers that are sending us EDNS0,
it's still pretty small, but it's growing, and it's something that we're encouraging.
Because it's better for Optimide to know who the actual clients are, so
that we can make a decision that's really based on the client rather than
the client's name server.
This is especially important for
clients that are using name servers that are not near to them in the network.
>> Right, like open DNS and Google DNS, possibly.
Right.
>> So, coming back to the discussion about core points if we were to summarize that
would it be fair to say that you are trying to trim this map of 300,000
prefixes to a more manageable ten or 20,000 or something on that order.
By ignoring the part behind the core point, right.
So you're trying to measure performance to the core point and
making the assumption that behind the core point the path is the same so
there is not much advantage to collecting the measurement there.
>> Well, >> That is almost right.
You are absolutely right.
We are looking for
these places where users are hanging on to the edge of the Internet.
And we don't want to have to map out to every user.
And we can't either.
If we started doing performance measurements to home users.
>> It would cause a lot of, a lot of uproar, okay,
it's intrusive, it might cost them money.
You can't really do that.
But, the point about why we stop at the core point.
It's not so much that we assume the rest of the path is the same, It's that.
>> The core point is chosen so that all of our paths intersect there anyway.
So it wouldn't matter which server we chose.
We couldn't have any impact on what happened afterwards.
>> I see, yes.
>> So, measuring beyond that point doesn't provide any useful information.
>> You know all these things that are beyond the same
exit number on the highway,
you're measuring the performance on the highway to each exit, in a sense.
>> Right, and from that point forward the routes the same, anyway, and
we can't do anything about it.
We can only choose the exit okay.
>> Cool.
>> All right?
>> Well so, there's an elephant not the room that I want to ask you about.
The elephant not in the room, being the hyperscale cloud data centers that
have not been part of this discussion at all.
And this is actually a course on cloud networking.
So why are we talking to, is Akamai a cloud service?
>> So you've go two questions there.
Why are you talking to me first, I may be the only one who was willing to talk.
>> We'll have to see.
>> You know, that's sort of like, you know, why are you looking for
your wedding ring over here when you know you dropped it over there.
Well, this is where the light's better.
But, he's after my cloud service.
I think the answer's yes.
In some ways, we're the unique cloud service.
Because we have a very widespread deployment of
rather than establishing a presence where we have
a small number of data centers, maybe a few dozen with thousands or
tens of thousands machines in each one.
We have thousands of location, but on average only, well,
we could do the numbers, but I don't know, 100 machines in each location,
maybe a little less than that.
So, where our deployment is a little bit different than the mega data center
type of cloud services that people might be familiar with.
But, on the other hand it is a cloud service in the sense that, we
offer to the content providers a service which is, we'll help deliver your content.
We'll provide the infrastructure.
You don't have to worry about the details of where it's happening or
how we get it done too much, and it's just something you outsource.
And in that senseI think it's a cloud service.
Another distinction is that
where cloud service were our customers or the content providers not the end users.
The end users don't pay us directly.
It's the content providers who pay us to help them provide better services.
To their end users so it's a little different then say,
you know other cloud services for example, if I'm at home and I do a Google Search,
I think I would say I'm taking advantage of a cloud service.
They're providing me with search results and it's done somewhere in the cloud.
But I'm an end user.
Or if I use Dropbox to store my files, that's a Cloud service, I'm an end user.
But Akamai's customer's aren't the end users.
We interact with them but the customer's are actually the content providers.
>> Fair enough.
>> So.
>> The role you have at does give you some freedom to think about the future.
So, one question about the future is do you
think that the hyper skill cloud providers.
>> We'll move into more of a kind of role that Akamai
has that instead of a small number of very large data centers,
a larger number of more geographically distributed small data centers.
You think that's a direction that other of the writers will go.
>> Or from your perspective,
do you foresee moving into providing general purpose compute at your locations?
>> Well, let's tackle the first question.
I think it's already happening right, there are other companies like Google.
Who has done the same sort of thing originally,
because of the great demand for YouTube video.
Google, not only has built their own backbone, but has also established servers
inside of many other networks, that they can serve their YouTube videos from.
And their deployment has grown very rapidly over the last three years.
To the point where, at least it's on sort of the same scale as optimize.
Maybe not quite as diverse, but it's fair to say, it's on the same scale.
And they're also using their servers now as a front end for
other content besides YouTube, such as for the search page.
Microsoft is moving in the same direction, Bing is moving in the same direction.
All three of these are cases where we have customers or former customers of ocomi,
that are so big, and have such tremendous content
demands that they have decided that they want to do some of this themselves.
They see the value in this kind of broad deployment, so that happens.
There is some movement towards doing that.
And it makes sense.
Others may do it, too.
As far Aqualie offering a general purpose computing platform,
it's hard for me to predict the future, but my understanding, and
I'm not an official spokesman about corporate strategy, but
my understanding is right now, we're really looking at.
Trying to satisfy the needs of our particular
customer base which are content providers.
Our customers are not the general public who want to rent resources
from us on a small scale to do their processing or their storage.
And so, although in the past, as far back as 1999 we did have an offering
where we could provide a virtual machine to whoever wanted it.
We discontinued that and I don't know about plans to go back in that direction.
But again, I'm not necessarily privy to all
strategic decisions that the company might make.
Any final questions for the Bruce?
>> I think we are close to time, but Bruce do you have any interesting or
exciting war story or final thoughts that you would like to share?
>> I do have some, I do have a few war stories.
>> Okay.
>> I have to do this very quickly though, so let me just tell you a very short one.
When I was head of engineering, one day we
got a frantic message from the network operation center that
one of our servers was reporting an incident in which it was receiving
a properly authenticated messages but the format was bad.
And so it was throwing up errors and our immediate concern was that somebody had
gotten hold of a private key of ours that could be used to sign messages.
And was now crafting control messages and trying to send them around to
different Akamai servers, perhaps in an effort to hijack the system.
And they were not quite right, the format wasn't quite right, but
it was It was really close.
And we didn't know how much time we had before this party alighted on the correct
format and then actually started to impact what the system was doing.
Well, this was the early days, so with a few engineers
we stayed up two nights in a row, trying to sort this out.
And then we finally realized it was just a server of ours that had fallen
off the rolls several months earlier and hadn't got any software updates.
And it was dutifully trying to rejoin the home system like V'ger in the first
Star Trek movie, just wanted to come home and we mistook it for an attacker.
And it got me to thinking that we don't really have any idea how many
servers have been abandoned out there.
Once you start deploying tens or hundreds of thousands of servers.
It's not just us, I'm sure Google and others have servers all over that are just
there like what happened, why am I not connected anymore?
But then the most embarrassing thing was after a little postmortem,
we realized it had been sending these messages for three months and
the alerts had been firing for three months and just nobody noticed.
>> [LAUGH] >> I could go on and
on with these kind of stories, they're fun.
There's a little lesson in each one I think.
Something there about, it's a little brittle to just
have a static configuration file that has a list of all your servers in it.
And if there's ever a deletion from that file, you've got a server that's going to
not get its software updates, that's not a good design.
And that's something we learned from that.
>> Yes.
>> Nice.
>> That's kind of similar to what [INAUDIBLE] ran
into [INAUDIBLE] in terms him having to fix everything in the world.
>> Right.
>> Right, right.
It seems much better to build in mechanisms for
dynamic configuration when possible.
>> All right, so I guess we'll about have to wrap it up there.
>> Sounds good, yeah.
Bruce, thank you so much for taking the time to talk to us and
this was very informative.
>> Thanks guys, thanks for inviting me in.
I hope all the students in the class are enjoying their time and
I think they should be lucky that they've got
such interesting instructors to teach this course.
>> Thank you.
>> Thanks a lot, Bruce, thanks for your time.
[MUSIC]
Explora nuestro catálogo
Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.
Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.
© 2019 Coursera Inc. Todos los derechos reservados.
