Routing Working Group
11 May 2017
9 a.m.
JOAO DAMAS: Good morning, let's get started. This is the Routing Working Group at RIPE 74. Hopefully that's the room you intended to be in when you came in. What we have up there on the slide is today's draft agenda.
Four presentations and sometime for further discussion if anyone has any. So let's get moving.
I'll start by thanking the scribes. I think Anand and his colleague will take care of scribing and Jabber, the stenographers, which make is possible for us to understand ‑‑ I actually never understood how we get anything done before we had stenography here.
If you have ‑‑ if you want to go to the microphone, please state your name and affiliation. This is mostly for the sake who is attending remotely since they cannot see you when you are at the microphone and it's also nice to be able to tell.
Minutes: We published the minutes from our previous meeting RIPE 73 back in December, I haven't heard anything about comments or notifications. Does anyone have any? No. Okay. So they become final. One last point I would like to ask people to do is rate the talks as we go through them. It is quite helpful for us in prepping for future agendas.
Okay. And just to end this part at the beginning, I would like to remind people of what our Chair selection process is. It's also on our web page and the reason I am refreshing everyone's memory about this is because I'd like people to start thinking about standing up to be chairs for the next meeting. I myself have been doing this for 157 years and that's a bloody long time. Sometimes it's just good to have someone else come in and think differently. 15 years is way too long.
So thank you and let's get started. The first talk is Erik.
ERIK BAIS: Morning everybody. For introductions, Erik Bais, I am the owner of A 2 B Internet. We're going to talk about moving, as a community effort, a couple of weeks ago we started the discussion at AMS‑IX, to see if we could get enough momentum in the community to say we need to have a new default config for our route servers. So that's where the ‑‑ that's basically where the whole topic came from.
A short introduction. A 2 B Internet and how we got here. We implemented new equipment in our network and the new equipment gave the options to do and try new things. So, playing around with the filtering, we noticed that on the VM Xs, playing around we found out that there were some serious leaks going on specifically through the route servers.
So this is where we are currently and you can see how your own Internet exchange is doing if you click on the link. Thanks for the maintainers of that info. That's Job and Sam err.
So, there's basically currently there is a match of whatever the various route servers are providing at the various Internet exchanges. Some are doing no filtering at all, some are doing just IRR data. Some are filtering on IRR data and ROAs or providing the options for that. And there's basically not a real standard.
And from a peering perspective for what you can actually expect from your route server, it's not helpful, because you actually want to say peer with the route server, and as a partial transit provider that the IXPs are currently these days, it's basically you want to have a solid set and you want to make sure that at least some confidence that what you are getting is accurate.
So, we want to get the community to agree on a new set, and I'll go through the presentation and I'll reiterate this at the end again. We want to have a new default for the configuration, specifically have the following sets in place.
So I'll go through it. There should be a normised configured Max prefix per peer. We all do this. On every peer that you have on the Internet exchange you set a Max prefix. Most people have their data registered in peering DB anyway. There is the small prefixes. You want to get those out. You don't want to accept any default, and you want to delete all the private and bow van ASNs. The list is not complete but there is some more further on. And an interesting one is basically, drop RPKI invalids on the route server. And that's actually and interesting one if you do this on the route servers even if you are not doing anything with the with RPKI yourself. Because it will actually give people an incentive to get their ROAs correct. So if people want to get their data across the Internet exchange instead of over the transit, they should update their ROAs, or don't have ROAs but at least you know correct the incorrect ones.
So that is actually giving your customers an incentive to do their part as well and actually fix the route objects and the ROAs that they use themselves.
So most of the data is already here. This is something we pull using small script per AS, you can basically get this data out of peering DB, it really is 7, 8 lines of code that you can do this. So the data is already there and the majority peers on the Internet exchanges have all their data already in peering DB. And this is basically where you know it's not really what can the Internet exchanges do for us, but it's also what can you do yourself.
So, and specifically for the smaller networks, you want to get some filtering on the peerings and want to do that yourself. But you need to keep a generic and effective if you don't want to do strict per peer prefix list. So there are some examples that we're going to use hereof suggestions that you can use, and this is very important.
So these filters, they are basically you know, how can you generalise this, and it's pretty easy with the examples that we give in the presentation and we'll also have them listed further down.
So this is basically how that looks. On the first one, there's been presentations on that initially. There is a list of AS numbers which you would typically not expect on any peering platform. You know, the majority of the parties here for you know the parties that are not on the list are not expected to have those as a peer. So on a route server, none of them actually appear in the route server and if they do it's a leaked route.
Now, that might actually not be true for everybody, because you might actually peer with you know some of the parties in this list. So you need to sanitise that and you need to actually go through the list for yourself and make sure that you are not filtering out any of your peers, but typically, you know, for the 95% in the room, this is probably the list that you can easily filter out. Those are all known transit providers, tier 1s, the usual.
Then we have the bog on ASN, this is typically how that would look. Prefix lengths range filter.
Too many AS path. So if you want to filter something out on the length of the AS path, I haven't seen anything in my configuration with more than 40. If you want to ‑‑ so if you want to have a safe line for that, make sure nobody gets, goes out of his way, put it to 100 and you can also, if you are using communities yourself, you can also filter at your peers, that they are not sending or you are not accepting any communities coming in from your peers or transit.
So, all these things is basically what we thought about, we thought you know maybe we should get things together and document them and that's how Job Snijders and I, we ‑‑ so I approached the NLNOG foundation, I said well I want to have a specific page where we can document all those filters and add them. I have done some on Junos, maybe somebody else is using it on Cisco, I want to add to them, so we have a specific page and guide made for BGP filtering. So, you can see the URL on the top. It's BGDp filter guide.nl NOG.net. And basically, it provides some basic examples why you would use or not use a specific filter. Configuration example for Cisco, for Juniper, maybe something else.
The interesting part is, as the whole side has been using GIT in the back, you can actually participate in this as well if you have good examples for Cisco or for Nokia or something else, please feel free to add that to the configuration if you want to.
So, this is ‑‑ you can see that here.
If you want to see the results that we're getting once you have the filters in place, this is the side that we have done on the client side, this is basically what you see here. The results here...
So these, this is for instance cogent and we can see level 3 in there, we can see G T T, and here again, there, and you can basically see safely assume that somebody here is, and it's not a single peer, it's all different peers as well, these routes that you see here on the left side, you can see that those routes, you know, might be old customers that they still allow in their prefix filter list for announcements or reannouncements. So we e‑mailed every party on the Internet exchange that was actually sending those to us, and the majority didn't even know. You know, something was not just updated and they said you know, nice catch.
And even somewhere intentional, because it was a customer and the customer just didn't update their filter yet but they did already re announce the prefix.
So we also had some interesting more specifics, you know, /28s or even smaller. You know, there is a tonne of /30 and /32s that you can see specifically on the route servers. And all that stuff should basically go away and you should basically filter that from a route server perspective but if your route server is not doing it, you should do that yourself as well.
So, there is the option obviously for strict peering. It's not too hard to actually you know generate filters if you want to do that on a per peer basis. Now, the configuration if you want to do that with a large route server, might actually become cumbersome. You know, you really need to automate this, because you don't want to go through prefix filters manually basically do this by hand, because then you would only have to rely on when peers are updating you that they have actually added more prefixes. You know, you see those e‑mails coming back and forth sometimes on mailing list, the majority of the people actually are not sending those, so you would miss out quite a lot.
And even then, it's not the Holy Grail. Sometimes you still get weird stuff. You know, people that actually create route objects for /30s, and that would still get sent over. So it's really you know you have to look at what is my peering policy, where does my routing policy and how do I want to do filtering.
So this is basically what we saw as well. You know, and it's a different question and everybody would probably see this differently. If you see that somebody is actually sending /30s, do you want to accept that or not, specifically if they are also having correct route objects created for that. So, I would ‑‑ you know, personally I would say filter everything out. But you know in some cases it would be valid for them in your specific use case to actually have that set up. Specifically if there are aggregated sub‑nets that you can find there as well. But you know, feel free to give some comments on that later because that might actually be interesting to see, and also to see how the IXP community can actually go forward or maybe that should be a configureable option. You know, what do you want to do with this?
So, and this is one of the examples that I just said. If you look at in this case the AS for the member list for AMS‑IX, they do a grab on /30, you can see there are quite a lot of very small route objects that will appear in that.
So, as a community, we should do better. You know, this is something from a couple of weeks ago and this is basically an issue. You know, there are a lot of hijacks, and although I do not condone on what they actually did here, both the parties that were you know, the reason for this, but also not the guys from SPAMHAUS, that they actually listed actively and started filtering on the mail server for DE‑CIX, that kind of behaviour is not done it does raise an interesting question. What the IXP, as, and how we see IXPs and how they need to behave with you know non behaving customers, how do we want to do this? Because for us as network operators, if we have customers that are doing something that it should not do, you know, we also have the, as network operators, we need to disconnect those customers, or at least restrict them and protect the rest of the Internet for you know their misconduct. So it's an interesting discussion that it at least brought it up, and specifically for the repeat offenders, you know, some of these guys are actually in this business of hijacking.
Specifically you know, there are some tools out there that list specifically and report on what's going on here, and some of them actually have been written about you know, block after a blog after blog about what some those customers are doing.
So, this is the last slide. This is the policy that we want to get through. Get the IRR data as the absolute minimum default filter. No default that should be accepted. Filter of Bogon AS numbers, RFC 1918 and the smaller more specific prefixes. Max prefix limit per peer, that would reduce the amount of impact after route leaks filter the known ASes. You know, the known transit ASes. Filter on RPKI invalids. And this might tee be an interesting one too that if we would start doing that, it actually you know sanitises a lot and it will actually give a good, you know, how do I say it ‑‑ a good incentive for customers to start updating their ROAs.
My suggestion would be to filter on small prefixes after acceptance of the IRR data filter. So that if there are valid route objects, then you can you know, filter always on the smaller prefixes as well if you ‑‑ and definitely lock down the outed hijacking operations, and you can basically lock them down with a manual configuration to their specific one or two prefixes that they actually should be announcing. If you want to keep them online.
Any questions?
AUDIENCE SPEAKER: Wolfgang Tremmel, DE‑CIX. Thanks for bringing up the subject of hijacking. I would like to ask the audience if you see hijacking on a BGP session, not necessarily on a route server session because we do the filtering according to route objects at DE‑CIX, and a lot of other exchanges do as well. If you see hijacking, gather evidence, let us know, we can only act if we have some evidence. Cannot act on pure hearsay. Thank you. Reduce
AUDIENCE SPEAKER: Ruediger Volk. Deutsche Telekom. One suggestion: When you do filters on the ingress side, actually try to generate reports for the sender about what you are dropping, and if you are really good, why you are dropping, which of your policies is violated by this specific root. Kind of as you said, sometimes people are not aware of the bugs that is in their configuration, and it would be a good idea to help them in fixing it.
ERIK BAIS: That's exactly what we did when we found out after we implemented some of the filters, you start looking at the hidden routes and that's a good place to start digging into.
RUDIGER VOLK: Kind of putting that I would suggest putting that into a description that your service description actually tells people you are going to do this on a permanent basis and that people, that people can expect this.
ERIK BAIS: Good suggestion.
AUDIENCE SPEAKER: And for say, turning on RPKI origin validation, yes, kind of first telling people what is going to be dropped next week is kind of a nicer thing than making them somehow sometime figure out something is not working any more.
ERIK BAIS: No. Obviously you know, for any Internet Exchange that is going to apply filters on route servers, but also on your own peering platform, on your peering router, you know, it's have some communication about this, this is not something you can just start doing.
RANDY BUSH: Could you go back one slide? The technical criteria and all the bullets under by default with fine except I have some qualms about prefix limit because there are those of us who carry traffic from regions of the world that like to have massive deaggregations Wednesday afternoon. And I have given up fighting it. My problem is adding the morals clause at the bottom. I think that brings us into a territory I don't want to get into arguments about and discussion. You know, spam, hijackers, people who serve bad could have, etc. I think that will take us down a discussion that has no proof of termination.
AUDIENCE SPEAKER: Marco duty tress. For I start I want to say that I totally agree with your proposal. I use it to report leaks at the Italian exchanges years ago but it did not do a lot. I just want to add a factual correction. At least according to the SPL record that is at page 18 SPAMHAUS did not actually list the DE‑CIX mid‑server. That kind of listing is a /32 on the dot zero address and it has the only purpose of communicating something. It does not add practical effects.
CHAIR: Anyone else? In that case. Thank you very much Erik the
(Applause)
ALEXANDER ASIMOV: Hello. I am from Qrator labs and today I'm going to provide you up with an update on the route leaks topic. This is not a new one, it was already presented here for a few times sometimes it was presented by me. But in case in this room there are newcomers and I hope there are here, I will start from the definition of route leaks and its consequences.
So, for the purpose of this topic I will speak only about transit route leaks. When a route leak is a situation when a prefix was learned from one provider peer and announced to another provider or peer. So the good question is: Why you should care. Why you should really care that for example one autonomous system accepted a prefix from one provider and announced to another one.
Unfortunately you should. Because at least these additional hops will increase your delays. These attacks could be also be done during the man in the middle, and do you really want to rely, to make your network dependent, your global connectivity, your availability on a network that already proved to be unmanaged?
So, let's see how networks are affected. This is the statistics was given in April and as you can see, there are thousands of route leaks on a daily basis, and from day to day, there are a set of affected networks exchanging, so during April we saw more than 40,000 different prefixes that were in route leaks. But in reality, the number of affected networks is higher, because a route leak is two sides wall. It's not only affects the leaks prefixes it also affects all networks that accept the leak prefixes, and what are these effects? You will be surprised but they are totally the same. So if you are accepting the leaks prefixes, you direct your traffic and traffic of your customers through these unmanaged networks with just the same results. And how many of our networks are affected by these? Nearly all of us. So, everyday, nearly each network accepts at least one route leak.
So, the problem is global. We are all affected. And who is here we could ask a good question. Who are the leakers? I will put out the scope the malicious leaks. They do exist but the majority of leaks is a result of lack of understanding and mistakes. But still, the number of the leakers or these autonomous systems who seem to have problems with their configuration and so on is really high. During April, there was more than 1,000 ISPs that were seen as the source of the leaks. And as you can see, this is not ‑‑ so if we will keep research we will find out that the number is even higher.
So, what we can do. Okay, we can try to reach thee guys, we can try to educate them. But if we have a goodwill, they may fix it or may not. So what we should focus, we should focus on the technical side of this question.
So what we have on the table now.
Of course we have communities. And if you set a proper community and after this set a proper future, you will prevent your network from making them. The problem here, if and proper. There is nobility in support in protocol, so, it's a common BGP problem, it's too flexible. And the result is that thousands of leaks. Another way of course is to build somehow filters to detect leaks and stop propagation. Maybe the best efforts here is to use AS sets, but here we have again a problem. Not all IRR support AS sets. More than not all AS sets are up to date. Not all AS sets are correct and you need no authorisation to for example to add in your AS set, AS list of customers. No authorisation process. Even if IRRs that support it. But okay, let's imagine ourselves in a perfect ideal world where all AS sets are up to date and correct.
Still, it will not solve general problem of route leaks, because if route leaks happens inside your AS cone, the source of the route will be correct and you have definitely accept it. And of course all upper level upstreams will also accept it too.
So, what is your last effort. Your last effort is monitoring. And this is the maybe the only option to detect route leaks that are taking place beyond your borders, and making some conexclusion.
Today we have, if we set up a proper communities, proper filtering, we would prevent our network from leaking. If we deploy some ingress filters, we can filter out some route leaks. And monitoring is the only option for you to detect a route leak that are happening with your prefixes and also maybe detect route leaks that you are accepting.
But, all these options do not provide you a mechanism to automatically, if you detect a route leak you are not able to fix it. From this point of view the problem seems to be very complicated. At the same time, if we speak about peering relationships, they are not complicated at all. We have only four of them. And maybe the fifth one which is some kind of flex which is a combination of these four. And from my standpoint, it seems like BGP lacks of native preparation inside of the protocol of these peering relationships. And so, to solve the problem, we propose to add BGP roles.
BGP role is new configuration option but just have these four value and it's negotiated at the start of the BGP session using open messages, so it's where you just send to your neighbour using capabilities and what does it mean if notification fails. It means that your neighbour or maybe you is trying to configure a BGP session in a wrong way. So, there is nothing to do. Just to drop the BGP session.
So, I think that roles are really native. Roles are not revealing anything to the third parties. So there is knowing to worry. And roles have a number of applications. So roles could be used to automate several previously manual configureable mechanisms.
Route leak prevention. As soon as we set up roles and we add one more tribute, which is called internal only to customer attribute, which has zero length, it's just a flag, and it is set on all routes that are learned from providers and peers. And we can also set automated filters that will filter out routes when we are announcing prefixes to other providers and peers and if this attribute is set.
So, I hope to see is just an automated version of communities without any fat fingers, without mistakes.
So here we can solve a problem of a leak prevention. But we can also solve a problem of a leak detection. So, please meet external only to customer attribute, which equals to the value of it has, it equals to the value of autonomous system who set it. So, if a route is announced to a customer, or peer, autonomous system should set it to ‑‑ its value to its own autonomous system number. These values should not be changed and so, these, in this scenario, it helps autonomous system 3 to detect a route leak that was made by autonomous number 2. It also seems to be very simple, it works.
So, what should we do if we detect route leak? It seems that we should filter it out, maybe drop the BGP session, but in reality, we should be very careful with level of our an aggression. Because detection is based on be attribute. A path attribute which is transitive and so it could be valued. And that is why instead of filtering, instead of broken sessions, we should only make allowance for local prefix value. That's all. Also, this is will normally solve the problem of a route leak.
So, we have already made an implementation using this, that you can find on the GitHub. And as you can see, this is not a lot of strings you need to configure your autonomous system to automatically prevent lake leaking and detect route leaks that are made by third parties. So, for me, it seems to be really cool. We have a solution for, a general solution for route leak problem. It is in code. There is no fat fingers in sight. It is fully automated. It is also verified by your neighbour, so if they have open messages granted that your role setting is correct. That's why we decided to move to IETF. But it proved to be an interesting but hard and long road.
I'd like to give here a special thanks to Randy Bush because without his help I think I would already give up. Today, we have two drafts. First one covers roles and IOT C. Second one covers E O TC. And I hope they will be finally adopted and we will see roles in in a release of software in the near future. There are also several Monday rabble mentions, one of them is BGP reject draft by Job Snijders which changes default behaviour of BGP router. So, if you will have empty ‑‑ not empty, absence of export or import policies, there'll be no exchange of announcements.
There is also a competitor of E O TC by other guys. And I'd like to highlight that the only document that has IFC status here is the informational document. That just describes what is route leak and gives us classification of types of route leaks.
So, here is a question: Should we blame IETF for these slow motion? You know, I see here, I think hundreds of people and it seems seems to be that you are all interesting in routing, I hope so, but I think not all of you are in the mailing list. So, instead of blaming IETF, collaborate with the IETF. This is the point.
And so, what we have on the table as a result.
In the meanwhile, you do not have any other options just to keep your community properly configure rated, keep your filters, monitor your prefixes to reduce the level of problems that route leaks create for your networks.
There is a chance that there'll be a change inside BGP protocol that will solve our general problem of route leaks.
But the existence or absence of this change depends on you and your collaboration with IETF. Thank you.
(Applause)
CHAIR: Thank you Alexander. Any questions or comments?
RANDY BUSH: There has been considerable discussion about the communities versus attributes, and it's to be noted I went around and asked questions of people who I know are using communities to try and scope enhancement by route servers and there have been noted and repeated failures of the scoping. People, you know, strip communities on input, ignore don't know export, etc., so that's why I strongly support the, making it an attribute.
GEOFF HUSTON: Age old problem, obviously. And BGP kind of is giving us two kinds of solution spaces, and when we last time around or the time before, or the time before that, talked about this idea that a single AS had a single consolidated routing policy with its peers for all the prefixes announced, I remember at the time when I said that being shouted down by others going no, no, no, you can be my customer and my provider simultaneously. You can't place that level of granularity. We have to do this per prefix. So, up came community attributes, you each prefix gets labelled. This seems to not work very well. So, we're back again. This AS has this consistent policy against its peers for all the prefixes it's announced. Now, I have no fixed religious opinion here and the pragmatist in me says if this has more traction let's give it a whirl because what we're doing before doesn't. It's not a new idea. It's the return of the other one and it's kind of okay.
What I worry, and you should worry, is that we'll flop back to per prefix and we'll flop back and flop back and never get anywhere. So the question to you and others is: Is it going to work better this time please? And is this going to really work? Because otherwise we'll be iterating between the two forever. So, offer me some assurance and I'm with you.
RANDY BUSH: Assure, assure, assure!! It worried me too. I don't like the complex relationship here. I think it's unnecessary and the reason is that in all cases I have found and I have done this myself in some large networks I once built, is when I have both a peering and a customer relationship with you, I actually run two separate BGP sessions.
ALEXANDER ASIMOV: So it's okay to have a different role setup on different BGP sessions between same autonomous system numbers. It's okay. It's no problem. And you don't need peer prefix any more.
GEOFF HUSTON: It is an old dog kind of thing and it's sort of close to a route leak the open attribute let's me knock it down, because I'm really close and I can see that's not a relationship I am expecting. If you want me to sit 5 AS hops away and not accept your route leaks, then I need a little bit better than that, because even if there are two EBGP sessions, with different attributes over there, by the time it comes to me, I don't see that and that's when you get into ‑‑
ALEXANDER ASIMOV: For this purpose, you can....
RANDY BUSH: Don't do it... I try to do it at a distance.
(Applause)
BEN MADDISON: Morning. My name is Ben mad son I am from work online communications and this is just a very brief update on the state of play with MANRS, which is the mutually agreed norms for routing security.
It's rather heartening that we spent the whole morning talking about routing security, and MANRS kind of seeks to address this from the other direction. So for those of you who are not familiar with what this is, I suspect that the number of people that applies to shrinking.
We know that there is a problem, we know that because we have been discussing it all morning and people have stayed so we can fairly easily skip over these things and we are aware that although there are tools and data structures that are available to help us deal with it, they are, at best, partial in their coverage and have various different problems with them.
One of the most important problems with our current approaches is that they suffer from the old game theory problem of the tragedy of the commons. My security is very much dependent on the behaviour of everyone else in this room and your security is dependent on my behaviour. And there are very few economic or practical incentives in place in the system that we currently operate that give people pay office for keeping their side of the street clean. We have heard an a lot about what we can do to protect ourselves on ingress. We have talked a bit about what we can do try and strengthen the system overall by improving protocols. But there isn't an obvious solution in place for this.
And that's what MANRS fundamentally seeks to address. And the way that it does that is by trying to turn it into a community problem. The idea is to articulate a baseline that's expected of network operators, publish that and for those people who think that it's a good idea and have already incorporated it into their practices, to say so publicly and encourage others to do so.
Those four pillars which are, as I say just a baseline and they are not everything that a sensible network operator should necessarily be doing are the following:
The first one is filtering. That's predominantly trying to keep your side of the street clean in terms of filtering towards the rest of the Internet and from your customers, those of you that provide transit to other people.
Making sure that the corollary is true as well in that it is difficult if not impossible for people to propagate traffic with incorrect source addresses across your network.
When both of the above fail, to make sure that you are contactable. That when Erik sees new that list, he has a phone number or an e‑mail address that someone is actually watching.
This is particularly important, given the lack of complete coverage that we have in today's system.
Global validation is the fourth and quasi‑optional one because it's the one for which we don't have a straightforward technological solution today. We have various means of publishing a routing policy that third‑party networks should expect to be subject to when sending us data. That can be in the form of import/export policy, aut‑nums in the registries, that can be in the form of written route policies that live on the Internet somewhere and hopefully easy to find. They each have their problems and they are usable to different degrees and for different purposes and certainly automating around this is hard so that is considered the optional one.
We have a fairly healthy list of members that have signed up to this and publicly state what had they're doing. This isn't a complete list by any means, this is what would fit on the slide. The difficulty that we're seeing that as you can see from here that's flattening off. And there are, you know, I'm happy to have conversations about trying to guess why that might be happening. I suspect part of it is because a lot of the low hanging fruit has been picked. I'm hesitant to speculate publicly but I'm happy to do so in the bar. But it is a difficulty. And one of the issues that we suffer from as the MANRS community is there is a lot of preaching to the choir going on. Outreach to people you know and agree with already is is easy. Outreach to people you don't know and who haven't been exposed to this community and others like it and don't necessarily know what they're doing is wrong is a much more difficult proposition and this is what we're trying to address as a community and what we need help with.
There are really three parallel approaches that we're trying to use at the moment. One of the difficulties is that it is fairly hard to articulate a financial business case around keeping one's own side of the street clean because the impact is not directly felt by the network that is misbehaving a lot of the time and trying to articulate a business case around why this is good for the operator is important and there is some work going on by the guys from ISOC and a few members of the community at the moment around that.
Also alongside we're trying to create a community where these issues can be discussed and where problems, when discovered, can be rectified, and that's been ongoing since the outset.
Those of you that were at Madrid might remember that we spent sometime writing up a best common operational practices document and that's now published and it's available that URL. And the next strand of work is trying to use that document as a baseline and develop training material and certification modules around it, because it's hard, it's unfamiliar for an a lot of people, it takes sometime to learn and there is almost never one answer that applies to more than one network.
And, again, this is something that is in its fairly early stages and if people in particular have experience in developing learning materials around technical content, and I'm looking at a few of you that I know that's the case for, any assistance that you are able to provide in that respect is obviously helpful.
We have been discussing for sometime what a MANRS IXP partner programme should look like and the degree to which there is overlap with other standards body and how that can be avoid sod there is minimum dupecation of effort. This is hopefully nearing its completion but it would be very it go to hear from the IXP operators that are in the room and at the meeting that I haven't already spoken to about this or you haven't already had a similar conversation with about what degree of involvement you would like in this and where you feel that you can be of assistance. And as I say hopefully we're going to have some documentation available in the relatively near future to try and formalise that.
And then of course I am presuming that there is a decent number of network operators in the room because it's odd that you are here if there are no operators. The most important thing that you can do is signing up. The fact that you're at the Routing Working Group listening to this content at a RIPE meeting suggests that you are probably, to some degree, familiar with this kind of stuff, and that you may very well be doing most or all of these good things in your network today without telling anyone.
All we need essentially is a few minutes of your time to tell everybody add to that list, try and work towards that critical mass that is required in order to try and build a stable community around this. And to articulate the same message to your peers and your customers that might not be so familiar with this.
If there is any questions you have on the process of signing up and what is required and what is not required and any misgivings you might have from a business perspective, then you are welcome to come and talk to me. And there's plenty of information available on that site and easy ways of getting into contact with most of us who are involved.
So that's the URL. That's where you should be starting off. I am fairly easy to locate most of the time if you'd like to talk to me and there's a bunch of other people in and arounded Meathing who have involved with this as well. So please come and find us. Thank you.
(Applause )
AUDIENCE SPEAKER: Randy Bush: This sounds more like a membership club than doing routing and it would be nice if your chart also had on it bad routing events over that time, and I suspect it's growing faster than your membership. So, something is disconnected here.
JOAO DAMAS: Yes, but wing we're missing in your slide number 9 you see that membership is flooding. Is what does victory look like? That's when you expect that all ASes would participate and most of the ASes don't matter anyway.
RANDY BUSH: Victory is a page with a lot of logos on it.
JOAO DAMAS: If those logos kind of then talk to their down streams, that's how you communicate. Cannot expect ‑‑
RANDY BUSH: By the way IIJ's logo is on that page. I didn't do it.
BEN MADDISON: I think that the bad routing information events metric is fairly hard to come by, and that was particle a preparation problem. One of the issues we have is the source of though events are not the same as the source of the ASes joining that list and it's not clear where the inter section of those two lines is, where we start finding that we're getting people who were once the source of the problem becoming the source of the solution, and there is ‑‑ I suspect there is some critical mass point and I don't know where that is.
RANDY BUSH: The same week that NTT and level 3 bragged about their MANRS joining, level 3 bought the bad routing from telecom Malaysia and blasted the world and NTT bought it from fibre at home in Bangladesh.
JOAO DAMAS: And shit will continue to happen, yes, but you have, or at least try to do something about it, right?
BEN MADDISON:
RANDY BUSH: I think this room is trying to do something about it. I think that room is a bunch of marketing garbage.
BEN MADDISON: The baseline actions are explicitly a baseline, they are a set of of minimum expected practices that are a good idea in all networks, depending on the nature of the network there are very much other things that people should be doing and unfortunately things that they can't necessarily protect against because they are too far away from the source of the problem. This is more than anything else for me, the importance of this is having a community of people who care about this sort of stuff in order to be able to discuss solutions.
AUDIENCE SPEAKER: Ruediger Volk: Well, okay, I think I agree for a long time with Randy on this being more on the marketing side than on the technical solution, and my point on the technical solution all the time is that you are talking about we want to define a line that is the minimum. Well, okay, actually you need to define and figure out the direction to move. You were making remarks that well, okay, unfortunately the solutions do not kind of fully work. And some of the statements kind of reinforce the illusion of some people that certain supposed solutions are actually working solutions. While, on the other hand, as far as I know, kind of making strong points of where the direction is things can be improved are, or rather dropped on the agenda like when you say the ‑‑ well, okay, kind of actually putting in there something that says if you have resources that can be registered and certified in RPKI, do so.
BEN MADDISON: That is specifically said in the BCOP document.
RUDIGER VOLK: Okay.
BEN MADDISON: This is also a very specific approach to the overall problem. This does not seek ‑‑ this is an approach that seeks to address the tragedy of the commons aspect of this. This is not necessarily a technical solution and this is also a ten minute slot.
RUDIGER VOLK: No, no ‑‑ well, okay, not talking about the ten minutes. It is not ‑‑ it is a solution for making global validation of data available, and you have somewhere the points where you say well, okay, we do not have reliable validated data. Well, okay, that's obviously a very important basic point and we have a solution well, okay, not available to everybody in all circumstance but available to many. For those where the circumstances are against it, well, okay, making strong statements that they should be enabled can help us to get beyond those circumstances.
GEOFF HUSTON: I'm still feeling the warm glow of that reassurance I got on the back from Randy and Ruediger. Only ten minutes, I'll try and be quick. Never mistake motion for progress. And unfortunately I think MANRS is in the motion category. This sort of awareness rating and so on tends to go around in circles, and the progress is kind of you're re‑stating things that we knew for a long time and it's not progress and I think a number of us is so fed up with this problem that it's not enough any more. You actually want to see something that says how do we get better traction? And you either say I'm going to educate the world and the entire world will been tired educateded in BGP announces people and life will be perfect or you are trying to say can I make the tool have sharper blades. Because if that's the way progress lies, then you know, that's the way progress lies, but I still sort of in my head when I see MANRS go it's just motion and I'm still not quite over that hump as a personal reaction. So let me just put that as a statement. I have used up my time thanks.
BEN MADDISON: I completely appreciate that and that's why I wanted to highlight the flattening out of that curve because it feels like we are back around to the top of another iteration of that circle. And that's kind of why I'm here. I think that if we are able to turn this, at the very least, into a forum where that engineering panacea that fixes this can be agreed upon by those that care about it, then we have a platform for a solution to be made available. One of the reasons, as you guys pointed out after the last talk, we have never agreed on an engineering solution is partly because there's always been competing camps that haven't really outside the context if it's standards bodies have had a place to sit and compare notes and fight until they agree.
AUDIENCE SPEAKER: Leslie Daigle. I want to make a couple of comments about just sort of general comments on the topic and some specific comments about the earlier questions, it sort of felt like the earlier questions and comments were directed to material that weren't particularly in this discussion, I felt like I had stumbled into somebody's longstanding argument as opposed to a discussion here. Picking on what Ben didn't say in his presentation seems like a little bit dirty pool since Ben was talking about a MANRS update and not everything he is doing with regard to routing security which is a rather larger space I happen to know. But I did want to follow up on one particular thing in how Ben presented this.
We can't lose site of the notion of community. This room may not need to be reminded of the fact that this is Internet working and that network operators have to play together. The general world of people who happen to have networking may have lost site of that and maybe the functional equivalent at this point of every householder who believes that they can actually be an individualist and completely independent of everybody who lives around them, which is of course complete fallacy. So, irrespective of what you think of the MANRS proposal itself, although do actually read it before critique it go. Irrespective of whether you think this is a cabal, do think about the fact of whether or when we get technical solution to improve writing security we still need to foster the community aspect of networking if we want to have a successful Internet.
JOAO DAMAS: Thank you.
(Applause)
Next up Julian:
JULIEN GAMBA: Good morning everyone. I am from the University of Strasburg. I want to talk to you about the fragmentation of the BGP routing table. To this is work with these people here.
So, I want to start with this number. 647,000. That is the amount of IP prefixes as of last week that were announced on the Internet and this number is growing. And as, you know, that is a problem because obviously a large amount of IP prefixes that means large amount of memory to store them, and most routers don't have that much memory to store prefixes. And when they run out of memory, at best they slow down. But some other it... in our work, when we looked at it we saw that well some of these prefixes are fragmenting so maybe we could aggregate them and announce the aggregate instead of the more specific prefixes.
Some other prefixes are redundant. Maybe when you have a lot of very specific prefixes and you announce the covering prefix, the less specific prefix, maybe you just remove the more specific prefix and don't lose any routing information.
So, what we did basically is this. First, we wanted to quantify the problem, try to understand how bad the fragmentation is, the redundancy on the routing table and try to investigate where, after all these years, there is still so much fragmentation of the routing table.
So quantify the problem we used this classification. We used four categories of prefixes, top, deaggregated, delegated and lonely prefixes. Just so it's very clear to everybody, I am going to use the example on the left to introduce the cases.
So, on the example we have three AS, AS 10, 20 and 30. AS 10 and 30 are in a peering relationship. And AS 10 is a provider of AS 20. Then you have a route collector, which is peering with AS 20. And as you can see, all the prefixes being announced in the network. So, we got four prefixes. The first one 10.1 /16 is covering two more specific prefixes, so we classify him as top.
Then you got 10.1.1 /24, which is covered by the top prefixes. And the original AS is the same as the covering prefix. So we consider it deaggregated.
Then the other one, 10.1.2 /24 is also covered by the top prefix, but it is dedicated to a client. It's announced by another AS. So it's not deaggregated it's delegated.
Then you got 10.2 /16 which does not overlap in any other prefix so we consider it a lonely prefix.
So, using this classification on real data. This is what we get. So, this is the evolution of this classification over the last 15 years. As you can see, the purple line is a deaggregated prefixes and it's creasing a lot over the last 15 years. We went from 18% so 36% nowadays. What is interesting here is that if you take the combined proportion of deaggregated and delegated prefixes it is constant. Meaning that the deaggregation is increasing but it's increasing at the same pace of the Internet. That's reassuring, we have a lot of deaggregation but it's not growing worse and worse every year.
With this classification, we don't have any idea about the size of the prefixes. And if we look at ‑‑ if we classify addresses instead of prefixes, we got a different story. So, we still have a lot of deaggregation, it's going worse every year. But the lonely and the top prefixes are to the so flat any more. We got a lot of moments between them and we got less and less lonely addresses in here. And that's something we weren't expecting at all.
Just to sum up a bit this part. So, we got more and more deaggregation and less and less lonely addresses, so there is queries some movements between the categories. We wanted to know whether some prefixes go from one category to another, or maybe there were just less new prefixs in a given category, or maybe that just stop being announced on the Internet. So we studied what the dynamics of our classification of the prefix in the Internet.
So we got this big scary graph with a lot of information in it. So, first you got the disks, they are called disks, which represent each grief prefix from top to bottom, it's delegated prefix, deaggregated prefixes, lonely and top. The size of the disks is actually the amount of prefix on a given category on a given year and the lines between the disks are the amount of prefixes going from one category to another between two years. And what is really interesting here is that you can see a lot of movements ‑‑ a lot of prefixes going from lonely to tag gated and then vice versa. That can happen when, for instance, you have a bunch of /24s and for some reason you want to announce the covering prefix, the prefix you had, but still announce the lonely prefixes which are now covered by the less specific prefix so they are classified as deaggregated and then a few years later for some other reason you stop announcing the current prefix, so they are classified as lonely again.
So we got a lot of movements. And if we look at the disappearances of the routing table. We wanted to know if some prefixs were stopped being announced. Maybe they were announced for some reason to prevent some hijacks or whatever. There is not an a lot of prefixes disappearing between two months but clearly the deaggregated prefixes tend to, they disappear more than other classes of prefixes. So, it seems to indicate that the deaggregated prefixes are used for some special purposes. And our assumption is was that these purpose Yours sincerely traffic engineering but maybe it's something else.
So we tried to detect the use of traffic engineering for not only deaggregated prefixes but for all prefixes. So we looked at the two most easy way to do traffic engineering in BGP, namely ISP prepending and selective announcements. Unfortunately forecast path spending we can't see much. There is a usage of AS path prepending, which is roughly the same here, all that we can say is that AS path prepending tends to be used more and more over the years, we were at 5% in 2002 and now we are around 15%. But we can't say any more about the deaggregated prefixes. If we look at selected announcements. That's very different.
It's clear that in this graph we have the deaggregated prefixes that tends to be much more selective announced that other classes of prefixes except for the lonely. So we got the lonely and the deaggregated prefixes that are much more equivalent than the other classes. So it tends to indicate some traffic engineering for those two class of prefixes.
Then what we wanted to know if, who is engineering traffic? Maybe some kind of AS tend to use more traffic engineering than others. So we classified the AS by business type using public data, which is interesting in that if you take the 30 largest transit providers on the Internet, every single one of them is announcing deaggregated prefixes. Some way more than others but they are all doing it at some point. And this was very new to us because we thought that most of the traffic engineering was happening on the edge of the Internet and not at the core.
But clearly they are using a lot of deaggregated prefixes and it may be for traffic engineering. Fact, 45% of the prefixes they announce are deaggregated for the 30 largest transit provider.
And they also announce a lot of lonely prefixes, 30% of the lonely prefixes. So next we wanted to know whether we could ‑‑ what we could gain from aggregating the lonely prefixes they announced. So we take the lonely prefixes that are not used for traffic engineering in our data and we tried to aggregate them as much as possible. And the gain nowadays would be of roughly 50,000 entries in the routing table, so there is clearly ‑‑ there would be clearly a gain by, if you aggregate as much as possible these lonely prefixes.
So, to sum up:
So we saw that there is a lot of deaggregation, but if you look at the data over the last 15 years you can see that it suggests increasing as fast as the Internet. So that's what reassuring. We have deaggregation but it's not going ‑‑ it's not going faster than the Internet. But, we saw that some AS split heavily the space by announcing a lot of lonely prefixes and as of today we still don't know why they do that. We don't know. Maybe it's for traffic engineering and from some of those it is for traffic engineering, but also it's maybe for some kind of security to prevent some kind of hijacks. We don't really know as of today so that remains an open question.
And I was pretty fast. So if you have any questions, now is the time.
JOAO DAMAS: Thank you.
(Applause)
AUDIENCE SPEAKER: Thomas. I just want to point out that one source of deaggregation could be the introduction of RPKI, because aggregates that could you initially announce as /14, now you have to deaggregate into /16 so that you can work.
JULIEN GAMBA: We didn't look at the RPKI or ‑‑
RUDIGER VOLK: I don't understand Thomas, but...
GEOFF HUSTON: I was never sure whether the, if you had a scaling problem in BGP was the absolute size of these tables, you know the number of entries or whether it was the dynamic update rate. Both present issues around scaling. The question that kind of sits behind your presentation that I'm begging to ask, so I'll ask it, is have you correlated the update rate? Do more specifics exhibit more instability in BGP than aggregates? And using your taxonomy of got lonely, aggregated and whatever other adjective you used, are there visible structural differences between the update behaviour of these. In other words, does the presence of a covering aggregate change the way in which dynamic announcements of the more specifics operate in BGP or something else? Because in some ways, the dynamic behaviour is more fascinating than the static table size.
JULIEN GAMBA: We looked at the amount of announcements for all the class of prefixes but we couldn't see any clear trend for a category more than others. So, just the input there.
GEOFF HUSTON: So you're saying, be careful here, that the probability of a prefix being the subject of an update or a withdrawal is independent of whether it's an aggregate or an uncovered route prefix.
RANDY BUSH: How about insufficiently interesting.
GEOFF HUSTON: I think it's interesting.
RANDY BUSH: In other words ‑‑
GEOFF HUSTON: The reason why I think it's a very interesting question and Randy is saying ‑‑
RANDY BUSH: I'm not saying the question is uninteresting. I'm saying the answer to your question is yes, he measured it. But the results were uninteresting. Not completely flat. Not zero. Just nothing ex saying exciting and I assure you we have spent a lot of time looking at update rights, not just for this project.
GEOFF HUSTON: The long term question in my head about this is over the same period that you are saying that the percentage of updates ‑‑ sorry, of more specifics is roughly 52, 52%, we all see this. But at the same time, out there on the reg of the network down in my little corner, and in most of the peers of both route views and RIS, the absolute number of updates is relatively constant, barring resets, and the number of prefixes that are the subject of updates, barring resets, is fixed. How could a routing table grow from 100,000 entries to 660 thousand, yet the dynamic behaviour remain fixed? That's a good yes. Is it something to do with deaggregation?
RANDY BUSH: So far we are studying it as a set reply subject and separate project and Julian has not been in it
RUDIGER VOLK: Julian, on your observation that somewhat surprisingly large transit providers, some athletes, did very much of deaggregation. I wonder have you checked whether that is a phenomenon that is to be localised to certain regions?
JULIEN GAMBA: We looked at ‑‑ yeah, we tried to plot in a map the level of deaggregation by country, by country of origin of the prefix, but we could not see ‑‑ we couldn't correlate much from the plot. In the ARIN region and RIPE region, much of the countries don't deaggregate that much as compared to the APNIC and AFRINIC. But we can really see a very high level of deaggregation on some countries and the very neighbour of these countries a very low level of deaggregation. So I don't think there is a correlation by the localisation of the prefix.
RUDIGER VOLK: Okay. Because my experience tells me kind of there are regions al cultures of how people are doing things, and in some regions I disagree with the culture, and I haven't pinned that down and not analysed whether there are actually reasons for the different culture. Like, if I'm in a region that has has scarce connectivity and people usually try to have lots of different upstream providers, you end up in a situation where you want to do weird traffic engineering, while in a plentiful region, well, okay, you don't have to play tricks. You may be still be misguideed to play tricks, but...
AUDIENCE SPEAKER: Francois Devienne from BORDER 6. Good job Julian, very interesting presentation. One quick clue about the reason for seeing all these deaggregated prefixes. What we see from our customers we see more and more corporates, companies asking for their own AS and their own address space and they have issues for inbound load balancing because they talk to only a few providers or let's say services provider like Cisco and so on and they need to load balance the inbound traffic. So they basically deaggregate their prefixes to make sure that they can get this inbound traffic for different paths. So one of the conclusions would I draw out of your presentation is that we need better traffic engineering using BGP for inbound traffic.
AUDIENCE SPEAKER: Nick Holt from Microsoft. There was a talk on Tuesday from the FAS L CDN guys, it was IPv6 centred but they explained how they were using what you describe as deaggregation a lot for traffic management and I'm wondering where the other CDNs that are using Anycast how big a factor that is. You mentioned traffic management was a major aspect there. And also you didn't mention any IPv6 today. Have you done a similar research into that or is it just v4?
JULIEN GAMBA: So, the data is just for v4. We looked at IPv6 prefixes but there is not much IPv6 prefixes being announced so we couldn't do the same research we did about IPv4, especially before 2011, we had like 98% of lonely prefixes I think. So people were just announcing stuff, and it does not overlap with anyone and everybody is happy. And after 2011, we see deaggregation raising already to maybe, if I remember correctly, it's 30%, as of today. But there is not much data so it doesn't tell us a lot of stuff. So we have to see in a few years how all this goes.
AUDIENCE SPEAKER: Interesting. A few questions. When you look at the deaggregated prefixes, are you checking the prefixes or also the attributes like AS path, because if path is different, there is different ‑‑ there is a reason why it is deaggregated.
JULIEN GAMBA: Just, when we placentaed the prefix, well the prefix and the AS origin, but when we looked at the aggregatability and traffic engineering, yes, weapon looked at the cast paths and all the attributes. Did I answer your question or...
AUDIENCE SPEAKER: Is the role data available?
JULIEN GAMBA: Yes, it's on the Internet, yeah. You can ‑‑
AUDIENCE SPEAKER: I mean structural data.
JULIEN GAMBA: You mean the code I used.
AUDIENCE SPEAKER: I understand all the data you used is available, but you organise it, structured it, so the question is, is it available in the format we can take.
JULIEN GAMBA: It's not but I can make it Open Source. I plan to redistribute the code some day when I clean it a bit. I can give you the code if you want and all the data, no problem.
AUDIENCE SPEAKER: And there was one more thing but I forget. So...
GEOFF HUSTON: I had enough time to go back to read what I wrote back in 2004 from a similar study. More specifics are four times for likely to be the subject of updates than aggregates. So although it's only 50% of the prefix table, it's about 80% of any daily update rates so more specific are noisier. The ones that are the noisiest in amongst this category is when you are doing what I call hole‑punching where the aggregate has an origin AS of one and the more specific has an origin specific of 2, where you are punching a hole through. They are the most likely in terms of the more specifics to actually exhibit instability, which is kind of curious, it's not what you'd expect because it's not traffic engineering. It's not saying origin moving path. It's actually hitting the hole and for some reason that makes BGP churn a lot more. There is certainly something there that we don't quite understand, but yes, more specifics are more noisier by a factor of 4.
JOAO DAMAS: At least they were. That was when, 2004?
GEOFF HUSTON: Well to be perfectly frank, it was from 2002 to 2014. So sorry if I got it wrong. It's a span, I looked at the last 10, 12 years at the time. Soing in much changes in BGP urgently.
JOAO DAMAS: Okay. Thank you very much Julien.
(Applause)
With that, we are done for today I just wish to remind everyone as I said before, when you are at the beam this sum of err bothered to depth think about chairing this session the next time and put your name forward.
(Coffee break)
LIVE CAPTIONING BY
MARY McKEON, RMR, CRR, CBC
DUBLIN, IRELAND.
