[MUSIC] Hi, today we are happy to be sitting down with Bruce Maggs. Bruce is the Pelham Wilder chair computer science at Duke University. And he's also Vice President for Research at Akamai. >> Thanks guys. >> So what kind of content is Akamai serving? Who's using Akamai? >> Well, there's a very diverse set of customers who use Akamai. There are tens of thousands of different websites that use Akamai to assist in delivering their content. There are a few major categories so one category you've probably all heard about is video, so the largest source of bytes delivered is video. So movies would be a good example of that, we have some customers that use Akamai to deliver movies. There's also live move events where there's action that's being captured in real time and immediately sent out over the internet. And occasionally those events can be very, very large. Another big category of byte served is software downloads. Software downloads are sometimes served when a user is explicitly clicking on a button on a web page to download something. But, oftentimes they're taking place in the background, and it's just some kind of software update program that goes and makes requests to Akamai to fetch the latest version without the user necessarily being involved. And these two categories of traffic are what we would call latency insensitive categories where there may be bulk data, but, there's some tolerance for a little delay at the beginning, whether it's a delay in a few seconds of buffering before the movie starts or maybe if a software download's going on in the background, you know, the user is not going to know whether it started one second earlier or later. There's an entirely different category, which is the latency sensitive traffic. And the best, maybe the most important example here is e-commerce websites. Websites where the content provider is offering a product, offering a service selling something online, and they want to have snappy performance. And, there have been a number of studies that show that, even small reductions in latency in the time it takes to get a response when you interact with a website, can greatly improve the experience for end users and measurably increase say stays. So that's one of the reasons why we fought so hard to get our servers deployed as close to the end users as possible. Although I will say that another reason for having this very widespread deployment is that for certain categories of traffic like video, or software downloads where you have to serve a lot of bytes, you also need to find a lot of bandwidth on the Internet. And that's another reason for a broad employment. I'm going to just give you one more category, although I could probably go on and on, but, increasingly websites who are subject to periodic attacks of various sorts, use the CDN as a front end to provide the scale to thwart denial of service attacks and to implement various kinds of sanity checks on user inputs to, to shield the content providers back-end servers from malicious traffic. An example in this category would be things like banks and government websites. >> Mm-hm. >> Exactly. >> So, you mention these two categories in particular and [INAUDIBLE] of service. But in regards to streaming movies and software downloads being one category, latency sensitive traffic, like serving websites being another category. How do the needs of these two kinds of applications differ? What do they want from Akamai, do different things for them? What changes? >> Well, I think if you look at the primary needs in the case of the latency sensitive traffic, It's, it's important that the user gets a response as quickly as possibly, which means, if we can, we will want to cache a copy of whatever object the end-users are going to require as close to the end-user as possible. >> So you care about seconds, hundreds of milliseconds, tens of milliseconds? Seconds? Individual milliseconds? >> Probably not individual milliseconds but, certainly hundreds of milliseconds and maybe tens of milliseconds. You want to see. Yeah, sort of along that scale. You want to shave 100 milliseconds off the page or maybe off each object off the page. That kinds of scale. The video traffic is a little different. There, you really want to provide a once you sort of determine the capacity of the channel. Between the end user and the internet. You just want to make sure you can sustain standing delivery of video at whatever was the suitable encoding bitrate for that video. >> Mm-hm. >> Now, one thing they hadn't mentioned was that There are also tricks that can be used to reduce latency, even in the case where the object to be served isn't captioned. I know that sounds contradictory or perhaps hard to see how a CDN could do that but if you like I could also talk a little bit about over [INAUDIBLE] absolutely. Yes. So there are multiple techniques I presume that go into making things work for objects that are not cacheable. We can discuss all of them, as many of them as you can tell us about. Overlay routing or perhaps connection proxying for TCP. An event. >> Yeah, so those two are, they're very closely related. And I don't know if the class knows a lot about the details of the TCP protocol, so I'm going to review something very quickly. And that is that when a client wants to make a connection, to a server, say, a web browser connecting to a web server. The client initiates a request to open up a TCP connection between the client and the server. And at the beginning, there's something called the TCP three way handshake where the client sends a packet, we call it sin packet, the server sends a sin act. And then the client sends back an act, and at that point, both parties can start sending data. But also, TCP has built into it sort of a period of time where the two try to figure out how fast can they send data without actually causing congestion. And it takes a little while for them to ramp up their speed to that sort of ideal rate. >> That's right. >> And all of this adds latency. And so one question is, is there any way to speed up the initial part of a TCP connection? One thing you could do is move the server closer to the browser. And the reason that helps is that the reason this initial phase in TCP is slow is that there are many round trips going back and forth. If you reduce the latency, those round trips happen more quickly. So one technique that employs and provides as a service for many content providers is that we act as their front end. So, even for content that's not cacheable. The end user's browser will contact, hopefully, a nearby Akamai server will Launch a new TCP connection with that server over a low latency interaction. Meanwhile Akamai, the Akamai server will have already opened up and maintained for some time. A persistent TCP connection back to the content provider. Which might be a much greater distance and much higher latency, but the initial TCP face has already been completed and so the end user doesn't see any penalty for that. So that's I guess what a kid called TTP proxy. Where we have the ability to sort of serve as a benevolent man in the middle, who helps terminate the TCP connection more quickly and then passes everything through. >> So each Akamai user facing server will then have a persistent connection to all of the content providers or those that are sort of cached in some way. >> Well not all content providers use this technique and it may be that unless there have been a connection from other users through that server to a contact provider before. It may not have set up the connection yet. And, furthermore, within any one data center, Akamai kind of partitions which servers are acting on behalf of which contact providers. So, it wouldn't really be an all-to-all work, every Akamai server's connected to every contact provider. That wouldn't be efficient. >> Three server will at a particular site. One server will kind of specialize in a certain subset of the content, and another server might specialize in another subset, and probably these will be overlapping in some way >> That's possible, but I think a better way to think of it is, let's say we have a cluster with 64 servers somewhere. So, we use a technique called consistent hashing to map the different content providers content onto the different servers. Any one server may be serving content for, you know potentially hundreds or thousands of different content providers. >> Right. >> But, if you look at any one content provider within a cluster they might be one or two or four depending on how popular that content provider's content is. One or two or four servers that are helping serve the content for that content. Now it is also possible that we might take a content provider's content, different parts of the page and split it across different servers. But actually we typically, unless there's a really good reason to do that, probably want an individual content provider to be served and have one server, the whole page. Or have several servers that are each capable of serving the whole page and all the objects, rather than splitting it across servers. >> Yeah, that makes sense. >> Now, Ankit you asked about TCP proxy. You also mentioned something called overlay. And, what I've just described TCP proxying is you can kind of view that as an overlay network in the sense that instead of having the browser connect directly to the content provider and using whatever path is selected by the internet routing protocols. We've introduced an intermediate point which is the CDM's front-end server that's now proxying the TCP connection. So, that's a A kind of simple example of an overlay. There's another example that Akamai uses. Between our server, our cache, which I've called the front end server, and the content provider's backing. So, I said that we maintain a persistent TCP connection back to the content provider. In many cases that's true, but we also we may try to find a better connection than the one that is the default path provided by the internet routing protocols. We will also look for one hop paths that go through another Akamai server located somewhere else. Which is then, let's say proxying the TCP connection. It's actually proxying an HTTP connection from our server back to the content provider. And we'll explore both the direct path as well as several one hop intermediate paths and look for the best one. Best meaning lowest latency, or lowest packet lost, or best bandwidth, depending on the application. >> Right, and by one hop you mean, one hop on the overlay so the part to this intermediary might be several network ops, but this one hop in the overlay sense. >> Yeah, that's right. It could be actually ten router hops. >> Right. >> But when I say one hop is, you go through one additional Akamai server. You could imagine having overlay paths that go through more, but as far as I know, we don't do that. >> But then the object that the user wants would be downloaded from the original content server. So maybe an intermediate Akamai server. That's over 1 hcp connection. And then at intermediate server streams the same object over another HTTP connection back to the first Akamai server that was originally hit by the web climb, is that right? >> Yeah. So there's sort of a series of connections and as I mentioned earlier for the direct connection from the Akamai server back to the content provider will have set up a persistent TCP connection. Actually a persistent HTTP connection will also do that between lots of pairs of Akamai servers. So that if you decide you want to pull something over an overlay connection, TCP is already set up and ready to go and you can just start pulling the navel. >> So, Bruce, is there any advantage to doing this at lower layers. For example, maybe I understood this as the intermediate node in this overlay determine if the front end's HTTP connection can start another HTTP connection, right? So this is at the application layer. So is there any advantage you see in the future for example for doing software routing? So you do the overlay at the routing layer, and your routing in software. >> Well I think there might be additional performance games that you could achieve if you had even finer grained control of the network. But as a company Akamai has tried to stay out of managing layers Below the transport layer. We don't operate our own backbone. You may have heard that companies like Goggle or Microsoft in addition to the many other things they do, they run their own network backbones. We don't do that. So [INAUDIBLE] does have routers here and there, but they're mostly used just so that we can peer with other networks, make connections to other networks with servers that are sitting right next to the routers. So our use of lower level technology is, it's very limited. And we almost never, if not possibly never, are actually doing routing. And so we've just avoided that because it would require a whole new branch of expertise that we would need and so far I don't think our services suffered from not doing it that way. >> But, you're doing overlay routing and that overlay routing it gives you more direct control that you can control downloads directly in form by measurements. >> That's right. In some sense, we are still designing our own routing algorithms. And we're using our own policies. And we're incorporating things that traditional Internet routing items don't incorporate when we make decisions. For example, we do continuous performance measurements. We make decisions about what overlay paths to use or which end users to serve out which servers. Based on those performance metrics. But we're doing that all at the application layer. None of that is happening down at the network layer or below. >> Right, so with thousands or tens of thousands of potential intermediate points, even if you're just doing one hop overlay routing. That's a big infrastructure, it's a lot of possible paths to explore. A lot of measurements that you could be taking about the network performances that changes over time. >> Well we don't really do all of them. I think, for example, we have servers all over the world, and if, let's say we have an Akamai server in Frankfurt, Germany and we have a content provider in Paris, France. And we want to know is there any path that's better than the direct path provided by the internet. Well, we are not going to explore a one hop overlay path where the intermediate hop is in the United States. We just know that in advance it doesn't make any sense. You wouldn't want to route the traffic across an ocean and back. We're not going to look at hops in India or hops in Japan. So we, there's a great pulling and sort of some sanity checks that go on when you prepare your list of potential candidates that might be used for the overlay. And furthermore, we collect history over time so we've seen in the past which intermediate hops set succeeded in providing better performance. And so probably, the number eligible hops might be on on the order of a few dozen, not thousands, like you can conceive of if every single server location that we maintain could be a candidate. >> So then when the request comes in, because you have this history, do you already know when the request comes in? You generally already know what the best path is going to be? Or are there additional probes that happen at that time? That's a good question. So on the one hand, I think that after you've done this for a while you should hesitate to ever say I already know what the best anything is. Okay because you never really know. And the internet changes very rapidly. And you just hope that you make a good decision as often as possible. But, having said that, you know one of the things we're trying to do is reduce latency. So the idea of waiting until you get the request, and then doing a bunch of measurements, that won't work. And furthermore, you because conditions can change pretty quickly on the internet, the routing paths can change, or even more quickly the congestion may come and go. You have to do kind of fairly continuous measurements on a periodic scale on the order of minutes or less. So they can look back and say hey, I just did a measurement and this pass seemed good but this one seemed poor. And so that's what we're doing for the overlay network. For example, between various pairs of optimized servers or may say server locations there's a cost continuous for sending packets back and forth, seeing what happens. But, you have some way of gauging whether the performance is good between that pair or whether that's a link you might want to use as part of an overlay routing path. >> And how big are the differences in performance if you pick sort of the default internet path, that's what most of us are doing. Versus a path that you've had some amount of control over because of the overlay routing. >> Well there's no simple answer to that. In some cases, it's the difference between the packets not getting through at all and the packets getting through. And that's the case where you really want to be sure you picked the right one. Other times it might be a difference of shaving off a little wait and see, or getting a little more bandwidth, and it's not as pronounced. But really what you want to avoid is cases where you can't even provide the service using the default path. >> And that would be because of some routing anomaly or a spike in congestion, or- >> Yes, those would be two good examples. >> So, the ping Akamai servers is one thing. There's a small number of, in comparison to the end credit large, there's a relatively small number of end points that you're considering for the overlay. What about when you're talking about routes to IP prefixes? Do you also do the same sort of exercise there, and if so, how do you steal that? >> Yeah, that's a good question. It is a much more difficult problem. What your asking about is how do we figure out when we get a request for an end user for content, which of our servers we should send a request to. And again, depending of what type of content that is, our goal would be, well we just got to. We've just got to have the lowest latency performance. Or we know we need to provide you know, a steady through quick for a video. What's more complicated here as you said is there are many more locations of users around the world that our roughly 2500 locations of servers. And another complication is that we only control one end, the server side. We don't control the client side. So, we have to do something different. We can't just do continuous probing between every client and in the set of candidate optimize servers. We don't do that. Instead what we do is we try to build out a map of the internet that captures both its topology but also its performance in real time. And at a very high level, it works something like this. We collect every day, all of the requests that are made for service to and in particular, we collect the DNS resolution requests that are sent to us. So, for example, we might have a customer. We'll just call them xyz.com. And I just made that up, so if there is a real xyz.com, it's a coincidence. And they've decided they're going to use Akamai to deliver some of their content. And they created a domain called ak.xyz.com, and whenever they have a URL that uses that domain name Ak.xyz.com, that's a URL where the request is going to come to ocomi and we'll look at the rest of the URL and we'll hand over an object. Well, what happens is when the browser goes to that webpage, either goes toward the URL that stars with ak.xyx.com or downloads a page that has some embedded objects whose URLs start with that domain. We'll receive a request from whatever resolving name server or local name server the browser's using will come into Akamai and say hey, can you resolve this name ak.xyz.com? That's where we have to make the decision about where we want to serve this user. Now, we may not know precisely which client is making the requests. We'll see the network address, the IP address, of the local resolving name server. More and more we may see something called the EDNS zero extension which a local name server may provide to us. Which is the slash 24 prefix that the user's in. And based on that, we want to decide where to serve the user. Sometimes it's really easy. It could be a no brainer. So, for example, if Akamai server at the University of Illinois and you guys in your office go to a website that's using Akamai's services, and you're using a campus name server, we're going to get that request and say oh, we're already on the campus. We'll serve them from the server there, okay? And that's what we like best. We like it to be an easy decision. And maybe half the traffic is served from those sort of no brainer decisions where we got a partnership with some organization. And it's sort of, unless there's a catastrophe, we always do the same thing because it's always the right choice. But we can't do that everywhere, so we also have a lot of servers in a large commercial co-location centers, where we're just one of many tenants, and we potentially could serve users from a lot of different places out of that co-location center. And the question is, given that all the users and all of these co-location centers, which ones should we serve from where? So I'm giving you a little background, now I'm going to try to say, how we do it? So, we collect the DNS requests that we received over the course of a day and we sort them, we actually sort them and we extract roughly speaking the top 300,000 name servers that sent us requests in the previous day, once it sent the most requests. And this'll be like 99 or maybe even more 99.5% of the requests. I don't remember the exact numbers but this is the idea. That we get a few hundred thousand name servers that are responsible for sending us most of the requests. Then what we do is, for each of those named servers, we use a utility, maybe the class assertive called Traceroute, to map out the path from maybe a few dozen Akamai servers to that named server. And we look at these paths from all these different Akamai server locations, and we find out where do they intersect? Where's the common point so that all servers, any one of our servers if we tried to send traffic to that name server would have to go through this point. We call that a core point. We identify a few tens of thousands of core points around the internet and we associate with each core point the name server set up behind it. The other side of the core point from the Akamai servers. And then throughout the day, each core point gets pinged perhaps from a few dozen or more optimized server locations to see what's our path like right now from these server locations to that core point? And we assume that any requests that we get from the name server should be optimize according to the best path to the core point put that name server spot. And here I'm assuming that, you know the name server is a good. Proxy for the end users that when we get a request from a name server, if we optimized what we think would be best for the name server then it's also best for the end user. >> So, you're using the name query performance optimization to decide how to ultimately deliver the web objects then >> Yeah, yeah. I think that is fair to say. In the earliest stage, before the EDNS serial extension, where the name server can now actually tell us something about the client. >> Right. >> We used to only have the nameserver's IP address. And so the only thing we could do is you know, just hope that the names server was a good representative, the path to the name server was a good representative path for the path to the actual clients that were using the names. >> Well is EDNS widely deployed now? I mean, it seems nice it gives you more, you don't have to use this proxy, that's an overloaded term here in this conversation, but proxy for the actual client. The name server, that's what you had to use, but now you can see the actual client prefix, IP prefix. If the NSC was there. So, is it widely deployed now? >> I think it's widely deployed in the sense that there are certain very heavily used name services like Google's name service, then it does deploy it, and we do accept the EDNS0 prefix from Google, and we act on it. In terms of percentage of local name servers that are sending us EDNS0, it's still pretty small, but it's growing, and it's something that we're encouraging. Because it's better for Optimide to know who the actual clients are, so that we can make a decision that's really based on the client rather than the client's name server. This is especially important for clients that are using name servers that are not near to them in the network. >> Right, like open DNS and Google DNS, possibly. Right. >> So, coming back to the discussion about core points if we were to summarize that would it be fair to say that you are trying to trim this map of 300,000 prefixes to a more manageable ten or 20,000 or something on that order. By ignoring the part behind the core point, right. So you're trying to measure performance to the core point and making the assumption that behind the core point the path is the same so there is not much advantage to collecting the measurement there. >> Well, >> That is almost right. You are absolutely right. We are looking for these places where users are hanging on to the edge of the Internet. And we don't want to have to map out to every user. And we can't either. If we started doing performance measurements to home users. >> It would cause a lot of, a lot of uproar, okay, it's intrusive, it might cost them money. You can't really do that. But, the point about why we stop at the core point. It's not so much that we assume the rest of the path is the same, It's that. >> The core point is chosen so that all of our paths intersect there anyway. So it wouldn't matter which server we chose. We couldn't have any impact on what happened afterwards. >> I see, yes. >> So, measuring beyond that point doesn't provide any useful information. >> You know all these things that are beyond the same exit number on the highway, you're measuring the performance on the highway to each exit, in a sense. >> Right, and from that point forward the routes the same, anyway, and we can't do anything about it. We can only choose the exit okay. >> Cool. >> All right? >> Well so, there's an elephant not the room that I want to ask you about. The elephant not in the room, being the hyperscale cloud data centers that have not been part of this discussion at all. And this is actually a course on cloud networking. So why are we talking to, is Akamai a cloud service? >> So you've go two questions there. Why are you talking to me first, I may be the only one who was willing to talk. >> We'll have to see. >> You know, that's sort of like, you know, why are you looking for your wedding ring over here when you know you dropped it over there. Well, this is where the light's better. But, he's after my cloud service. I think the answer's yes. In some ways, we're the unique cloud service. Because we have a very widespread deployment of rather than establishing a presence where we have a small number of data centers, maybe a few dozen with thousands or tens of thousands machines in each one. We have thousands of location, but on average only, well, we could do the numbers, but I don't know, 100 machines in each location, maybe a little less than that. So, where our deployment is a little bit different than the mega data center type of cloud services that people might be familiar with. But, on the other hand it is a cloud service in the sense that, we offer to the content providers a service which is, we'll help deliver your content. We'll provide the infrastructure. You don't have to worry about the details of where it's happening or how we get it done too much, and it's just something you outsource. And in that senseI think it's a cloud service. Another distinction is that where cloud service were our customers or the content providers not the end users. The end users don't pay us directly. It's the content providers who pay us to help them provide better services. To their end users so it's a little different then say, you know other cloud services for example, if I'm at home and I do a Google Search, I think I would say I'm taking advantage of a cloud service. They're providing me with search results and it's done somewhere in the cloud. But I'm an end user. Or if I use Dropbox to store my files, that's a Cloud service, I'm an end user. But Akamai's customer's aren't the end users. We interact with them but the customer's are actually the content providers. >> Fair enough. >> So. >> The role you have at does give you some freedom to think about the future. So, one question about the future is do you think that the hyper skill cloud providers. >> We'll move into more of a kind of role that Akamai has that instead of a small number of very large data centers, a larger number of more geographically distributed small data centers. You think that's a direction that other of the writers will go. >> Or from your perspective, do you foresee moving into providing general purpose compute at your locations? >> Well, let's tackle the first question. I think it's already happening right, there are other companies like Google. Who has done the same sort of thing originally, because of the great demand for YouTube video. Google, not only has built their own backbone, but has also established servers inside of many other networks, that they can serve their YouTube videos from. And their deployment has grown very rapidly over the last three years. To the point where, at least it's on sort of the same scale as optimize. Maybe not quite as diverse, but it's fair to say, it's on the same scale. And they're also using their servers now as a front end for other content besides YouTube, such as for the search page. Microsoft is moving in the same direction, Bing is moving in the same direction. All three of these are cases where we have customers or former customers of ocomi, that are so big, and have such tremendous content demands that they have decided that they want to do some of this themselves. They see the value in this kind of broad deployment, so that happens. There is some movement towards doing that. And it makes sense. Others may do it, too. As far Aqualie offering a general purpose computing platform, it's hard for me to predict the future, but my understanding, and I'm not an official spokesman about corporate strategy, but my understanding is right now, we're really looking at. Trying to satisfy the needs of our particular customer base which are content providers. Our customers are not the general public who want to rent resources from us on a small scale to do their processing or their storage. And so, although in the past, as far back as 1999 we did have an offering where we could provide a virtual machine to whoever wanted it. We discontinued that and I don't know about plans to go back in that direction. But again, I'm not necessarily privy to all strategic decisions that the company might make. Any final questions for the Bruce? >> I think we are close to time, but Bruce do you have any interesting or exciting war story or final thoughts that you would like to share? >> I do have some, I do have a few war stories. >> Okay. >> I have to do this very quickly though, so let me just tell you a very short one. When I was head of engineering, one day we got a frantic message from the network operation center that one of our servers was reporting an incident in which it was receiving a properly authenticated messages but the format was bad. And so it was throwing up errors and our immediate concern was that somebody had gotten hold of a private key of ours that could be used to sign messages. And was now crafting control messages and trying to send them around to different Akamai servers, perhaps in an effort to hijack the system. And they were not quite right, the format wasn't quite right, but it was It was really close. And we didn't know how much time we had before this party alighted on the correct format and then actually started to impact what the system was doing. Well, this was the early days, so with a few engineers we stayed up two nights in a row, trying to sort this out. And then we finally realized it was just a server of ours that had fallen off the rolls several months earlier and hadn't got any software updates. And it was dutifully trying to rejoin the home system like V'ger in the first Star Trek movie, just wanted to come home and we mistook it for an attacker. And it got me to thinking that we don't really have any idea how many servers have been abandoned out there. Once you start deploying tens or hundreds of thousands of servers. It's not just us, I'm sure Google and others have servers all over that are just there like what happened, why am I not connected anymore? But then the most embarrassing thing was after a little postmortem, we realized it had been sending these messages for three months and the alerts had been firing for three months and just nobody noticed. >> [LAUGH] >> I could go on and on with these kind of stories, they're fun. There's a little lesson in each one I think. Something there about, it's a little brittle to just have a static configuration file that has a list of all your servers in it. And if there's ever a deletion from that file, you've got a server that's going to not get its software updates, that's not a good design. And that's something we learned from that. >> Yes. >> Nice. >> That's kind of similar to what [INAUDIBLE] ran into [INAUDIBLE] in terms him having to fix everything in the world. >> Right. >> Right, right. It seems much better to build in mechanisms for dynamic configuration when possible. >> All right, so I guess we'll about have to wrap it up there. >> Sounds good, yeah. Bruce, thank you so much for taking the time to talk to us and this was very informative. >> Thanks guys, thanks for inviting me in. I hope all the students in the class are enjoying their time and I think they should be lucky that they've got such interesting instructors to teach this course. >> Thank you. >> Thanks a lot, Bruce, thanks for your time. [MUSIC]
