Plenary session
12 May 2017
11 a.m.
BRIAN NISBET: Hello. Good morning, lovely RIPE 74 attendees. So, we have come to the very last session which I will not be Chair, this is an optical illusion. However, before we get to the last session, there is one more set of election results to announce. And I, as an apparently neutral party, will be announcing them.
So, we had the PC election and thank you very much to everyone who voted in that. The ongoing kind of the ongoing building of the PC is hugely in getting the voices in there and the different points of view, and obviously, you know, it feeds into what we were talking about in regards to diversity this morning. So we had three candidates and thank you to Benno, Franziska and Sean for putting themselves forward for all of us, and the results of the election are that Benno and Franziska were elected to the PC, so thank you very much for that.
(Applause)
So, Benno continues as our overlord and leader. Thank you very much, as obviously Sean for putting himself forward. For those of you who didn't get the opportunity this time, there will be more elections. This is an ongoing process. So thank you very much. I'll hand over to Leslie and Benno to chair this session.
CHAIR: And thank you everyone for showing your wonderful faces this morning, and thank you for Christoph for telling us more about BGP Flowspec and it's interoperability.
CHRISTOPH LOIBL: Hello. I was talking about BGP Flowspec roughly a year ago in Bucharest just across the corner around a question that was asked in the discussion that followed my talk was actually about interoperability of BGP Flowspec ‑‑ so, I thought of a very good question. So, we gave it a try.
So today I will talk about the BGP FlowSpec inter ability lab that we built. I'm Christoph Loibl I'm with next layer communications, but we also do DDoS mitigation for our customers and we are actually using BGP Flowspec in our network for like 2013 or so.
Actually, the work I'm going to present is the result of a joint research project together with Martin backer from T‑Mobile. We had very good support by the manufacturers. So, a special thanks to Nokia and Cisco who actually provided additional hardware for our lab. And I'm not going to suggest to buy this or that equipment for BGP Flowspec because I have been asked this question several times already.
Who in the audience knows what BGP Flowspec actually is? Hands up. Quite a few. And who is actually using Flowspec in the network? Less... obviously.
And are there any brave people who are actually exchanging BGP through routes with their external internal peers, with their customers, with their upstreams? At least one. I mean, to say with the words of one of my colleagues who always says you can't buy brave re. Actually, a very short introduction, what is BGP Flowspec. It's actually a network layer topology information that was defined in our C 5575 that allows us to exchange flow filters via BGP. So, we call, we actually call them flow components that can be like source destination IP addresses, source destination ports, ICMP types and so on so it's a whole lot of different flow components out there, they are defined in the RFC. And once a packet matches those flow criteria, it can be subject to a flow action. And those flow actions are encoded into extended communities on the BGP level. And such flow actions can be in rate limit, those flows to a certain amount of bandwidth or redirect the flow in a VRF. And those resulting policies, they are pushed to the routers via BGP so it's going very, very fast and the resulting policies can then be applied to the interfaces of the routers.
Since we are using BGP for this, it's capable of doing like distributing those access control lists not only within a single AS, but it's BGP, we just enable another address family and so we can, like exchange those filters also with external peers like with customers, with transits or whatever.
So, let me give you a very short example. It took me quite a while to find out this example IP address here, 123, there is some very simple network, some legitimate traffic is going on and at some point there is a denial service attack coming in and it's filling up all the links. There are several ways to mitigate such an attack. Would be remote blackholing, but this is ‑‑ yeah, customer is not really happy with this service provider has no problem, but our approach with Flowspec is much more elegant because we can push flow routes to the entire network and modify the forwarding behaviour of the network, so we can change the network to, for example, the following. We already identified, okay, a part of our attack is a very simple anti‑P reflection attack. We don't need to forward those packets anywhere, where he just drop right at the border of the network. For the rest of the traffic, with we may decide to forward it somewhere to a scrubbing centre, somewhere in the network, which just cleans the traffic and forwards the rest of the traffic to the customer. So customer is happy, network is happy, everything is fine.
One thing that's nice here is, we are not only protecting our own network in that case. We can ‑‑ we distribute this flow routes also to our neighbouring upstream AS, this 65501 and can ask them to do some dropping or some QoS classification for us already. So we are actually protecting also this interconnection between our AS and the upstream AS.
Well, this is the peer 3 and in the live Internet you are hardly ever see this Flowspec route exchange between external neighbours. So, this is actually where our lab comes in. We wanted to produce a working set of configuration for an inter AS Flowspec deployment because it's currently hardly seen in the Internet, so there may be a reason for this. We wanted to verify the behaviour of the different products because we obviously the Internet you don't have just one vendor in your network. Are they able to successfully exchange flow routes or can they talk to each other. We wanted to identify missing features, and last but not least, actually, we wanted to encourage our customers and peers to use BGP Flowspec and exchange filter rules with us.
One additional note is the lab that we were building was just targeted to control plane. We were just interested in the BGP signalling. We're not looking at the data plane and at the resulting filters in the hardware of the boxes.
So, this is actually the lab that we ended up with, you don't need to get all the numbers in the last row. So we had those four different vendors that we selected. We knew there is a decent install base at least in Austria for those different vendors, that's why we selected them to Nova, Cisco, Juniper and Huawei and realso decided to use Open Source free BGP implementation. We used exit BGP in this chase to have some more routers in the network. Every of this hardware boxes of the blue routers were in a single AS and we connected them more or less in a full mesh, so every router exchanged flow routes with all the other routers. Our plan was to actually announce the flow routes from this R11, R 12 which are on the very right and very left side.
Actually on paper things like this look very nice, once you plug everything in the lab, it's not that funny any more, it's like many many cables all around. Actually I put in this slide because this is the only picture of Martin that I captured during our lab sessions.
We also came up with plenty of test cases. One general match pattern, so we wanted to find out if all the vendors actually understand all the possible flow components that are in the RFC.
So, we also wanted to look into the action communities, so, we could, and combination we could have rate limiting and redirect into a rear effort at the same time, so we wanted to look into the capabilities of the routers there.
And transitivity of the action communities because they are sensing something a little bit fishy in the RFC.
There is policy frameworks. This is more or less vital to any Internet deployment because we are running off of routing policies, so we also need routing policies for BGP Flowspec.
Validation, there is some validation of flow routes built into the protocol. You can't just announce any flow route. You also need to IP route for that destination, otherwise that route won't get accepted. Term ordering. IPv6 Flowspec, VRF Flowspec. We wanted to get a general impression of what's there already.
So this is actually the first thing that we did. This is the BGP configuration of a float consisting of all the possible flow components that are available out there. Actually, this this flow route won't match any packet because there are ports in there, there are ICMP types, there, TCP flags, there is no packet with ICMP type, and TCP flage in the same packet, so this won't match a pattern but at the same time it's from the syntax, it's a correct update message what we will be sending to the router.
So, back to the lab. This is, I think this is not the PowerPoint, but anyway. We loaded the configuration into our R11 which can not be seen on this screen here. And next thing that happened was just the lab just literally exploded. We saw, like, hundreds, thousands of syslog messages going on. All the BGP sessions, more or less all the BGP sessions just started to randomly flap so there was no stable situation at all.
Only when we withdrew this flow specification again from R11, the lab somehow recovered. As a network engineer, when you face such a situation, what are you going to do? Any ideas? Try again. And have the same result. Yeah, what we did actually was we loaded our packet captures into TCP dump, or wire shark in our case, and looked at the packet. So, actually I didn't tell you the whole truth about our lab. We had like one single layer 2 switch in the middle of the whole thing and we had all the messages, all the signalling going through that layer 2 switch so we had like a synchronised packet capture of everything that was going on in the lab.
Anyone who sees what's the problem here? It's already highlighted, but so, Wireshark complained about the malformed packet and unfortunately it didn't tell us where the problem lies and since it was a really huge flow filter that we were loading into the ‑‑ we were sending into the routers, it took us quite a while to like manually dissect this message, it was like, I don't know, 200, 300 bytes or so, so we went manually over each byte and dissected the message and we were pretty close to find the bug, but we didn't find the bug in the BGP implementation. It was actually a bug in Wireshark, the first bug that we saw in our lab. It turns out that there are different ways of encoding this BGP messaging to the packet since we were having this huge flow specification filter, we needed to use this extended length encoding and there was a bug in Wireshark. By the way, it was really a pleasure to work together with Wireshark people because like in one or two days, we had a working version of Wireshark that was perfectly able to dissect the message.
So, you may say this, fixing the bug in Wireshark doesn't fix the network problem. The BGP will still be unstable and obviously this is true. So, I will go very quickly through a few examples, I am covering for because I want to be a little bit fair to all the vendors. What we saw in the lab. Actually on the last slide I have a URL to our paper that we compiled and it's like really huge and there are all the details about the testing that we performed.
Again, now we have the problem with the ‑‑ we actually announced a flow specification route from here to the Cisco router, and what happened next was that basically, those two BGP sessions started to flap.
Notice that this one session is not even close to where we originate our Flowspec route so it's like traversing two ASes because it's actually hitting or making a problem in the network. And what we saw that the Juniper guys just read the RFC a rely bit different because it's a little bit unclear in that manner and they said networks such a flow is not valid and just dropping the session with a notification.
So, next example we announce a Flowspec route from here, from to the Juniper router and what happens next is that all those sessions to the Cisco are starting to flap constantly. It turns out that the Juniper router correctly understood our flow specification but needed to propagate the flow specification it just did it wrong and it just really generated a malformed packet. By the way most of the things that I am presenting here are already fixed in the latest releases but I'm very sure that they are still out there in the network.
Next thing is we announcing this ‑‑ maybe I forgot one of the arrows, anyway we are announcing a Flowspec from here and it gets pretty much distributed over the entire network, but when it comes to the Nokia box, it sometimes gets propagated by the Nokia, sometimes it doesn't, and what happens here is there is a race condition built into the protocol itself, and most of the vendors somehow programme around this race condition but Acotel didn't.
Last example, Flowspec announcement to the Huawei box, but just nothing happens and it turns out that once the Huawei box is unable to programme the flow into the FIB, it's not propagating it at all. This is a good idea for IP routing, because when I'm not able to forward the packets, I shouldn't announce the route. But for Flowspec it's a different story.
So, I'm skipping this because of time.
Test summary. We actually found a whole lot of bugs, but we were not looking for bugs actually, we wanted to come up with a usable configuration for BGP Flowspec which, by the way we didn't come up with. We found a whole lot of different interpretations of the RFC, and this was ranging from unpredictable propagation of Flowspec to BGP flaps, and you don't want to see the BGP flaps in your live network. And we actually discussed all the bugs with the vendors and they were really cooperative. Even when we suggested changes that are not really covered in the RFC.
Missing features.
BGP import/export policies. You want to have this. You want to know what you're accepting from your external peers at least. So this is one thing that's more or less completely missing. You want to match on all the different flow components. You want to do this. Without this you don't exchange routes with external neighbours.
Flowspec for IPv6 is a completely different story. It's just a draft. It died somehow, then it was revived for the last IETF meeting. Now no one is working on it anyway.
And Flowspec for IPv4 in a VRF, it's actually already in the RFC, in the Flowspec RFC, but the coverage in the ‑‑ on the vendor side is really poor.
Conclusion:
Testing took longer than expected, or we didn't expect this. We saw incompatible NLRI encoding. As I told you there is this absence of BGP import/export filters that we actually cannot accept or cannot use for external BGP. And we saw a few unclear sections in the RFC 5575. So if you really exchange BGP Flowspec routes with your external neighbours, be careful. I'm not saying that it's impossible, but you really need to know what you do, assess your risks, because there are still a few problems out there.
There is some ongoing work. We are actually recently ‑‑ we recently published a draft that's meant as an update for RFC 5575. We wanted to, like, clarify all the unclear sections. We really find all the flow action communities as transitive because there was something in the RFC that was unclear but all the implementations implemented as transitive. We added a whole new section about flow action interference. It's easily possible that you have two actions that interfere with each other in a flow specification like redirect in VRF A and redirect in VRF B, so what should happen. We added additional action that we call the traffic rate packets. Currently it's only possible rate limits in bits per second or bytes per second actually. And this draft has already been adopted by the IETF into the main Working Group. We hope we can work on this further. And we recently see patches in go BGP and ExaBGP coming up.
These are the two URLs that I promised. One is our Flowspec paper. It's very ‑‑ like ‑‑ decent. We have all the data in there, all the packet captures, the configurations, you can read through it. We are really happy to get feedback on both on the draft and on the Flowspec paper.
So, thank you very much for listening. I am happy to take your questions.
(Applause)
AUDIENCE SPEAKER: Peter Hessler. In your comment about missing abilities to do filtering, are you referring to filtering within the Flowspec specific attributes or are you talking about accepting the Flowspecs in general for example a prefix that may be out of range?
CHRISTOPH LOIBL: Do I understand the question right? You mean those policies I was talking about?
PETER HESSLER: Correct, yes.
CHRISTOPH LOIBL: Actually what's very easy because we do have Flowspec consists of those two things. One is the LIR, one is the action communities, which are the extended communities. I say easy to filter on the communities because in most implementation it's there. I'm not a hundred percent sure if it's possible with the Cisco implementation currently, but with most of the implementation it's possible. But I want to filter on the flow components on warts of the NLRI. I want to filter on the destination prefix. I want to filter on certain components that are in the NLRI. And this is not possible in the implementations that I know of.
AUDIENCE SPEAKER: Okay. That's exactly the problem why we have not yet implemented Flowspec in open BGPd and we don't want to go forward with implementing if until its possible to filter and protect our network from a potentially hostile neighbour. Thank you.
AUDIENCE SPEAKER: Blake Zayo. Thanks very much for putting this together. I think most folks that use Flowspec were really looking forward to seeing something like that. Have you talked to the folks at Arbor about this at all and if not I'm sure they have somebody here you might have a chat with?
CHRISTOPH LOIBL: Actually, we haven't Arbor here in the lab, but they are very conservative what they are actually announcing. So, I think they don't even allow you to announce, to use all the different flow components. So, we didn't actually see a problem with Arbor currently, so, it works so far ‑‑ what they are doing is working fine because they are just using a subset of what's actually possible.
AUDIENCE SPEAKER: Daniel Karrenberg. Thank you, thank you, thank you. I really like this talk because it's what the RIPE should be about. People trying stuff out and then being totally brutally honest about what they found. I really like that.
(Applause)
Now, of course I have to ask a question. And my question is, you know, when I see this and especially interaction it looks like a huge canon to shoot various body parts with, so, I understand that your interest was to just put it together and see whether it actually even works, and I'm surprised that you were surprised on what you found. But, is there any work going on to basically do programmic configuration of this so that the shooting doesn't happen? So, like, you know, what I would like to see is a lab where you actually get people operating different ASes together and say how do we configure that so that we avoid this complexity actually doing more harm than good?
CHRISTOPH LOIBL: Actually, that's what we started off with. We wanted to have a configuration that's working. The thing is with the current tools that we have at hand, it's not possible. We can't do proper filtering. The incompatibilities that we see currently in the decoding of the NLRI, even if there were decent routing policies available in the configuration of the routers, they will still drop the BGP session because what ‑‑ many of the things that we observed were just happening in the BGP places when they just dissect the message. We can't even ‑‑ if there is a filter, it won't help us there. It will just break the whole thing again. So, actually we wanted to come up with a configuration for the entire lab that's secure and does what we actually want, but it turns out that there was no way.
DANIEL KARRENBERG: Thank you for being honest.
AUDIENCE SPEAKER: Alex
Semenyaka, RIPE NCC. I have not a question but a comment about your questions, who is using Flowspec. I think that the main issue we have here is that for procurement team, it's like we are spending some extra money to upgrade the equipment to support Flowspec to get less money for traffic from our customers. That's a real issue. And I think that this is the main issue with the deployment of Flowspec globally, so what I really want to hear next time is sharing experience with this part.
CHAIR: Thank you. Thank you again, Christoph.
(Applause)
Very interesting presentation. I really liked it. So, we are closing the Plenary part and now opening the lightning talks part. Can I invite Filiz for an ASO report update. So, we might like an NRO session today, but luckily we found Filiz volunteered to give this important update.
FILIZ YILMAZ: Thank you. Thank you, I am the ASO AC Chair. At the same time I am a RIPE selected representative in that Council so we thought we would give you an update. It has been a while. You elect us to this group, so thanks to the RIPE database PC, they managed to find that ten minutes for us as a lightning talk although we understand it kills the spirit of lightning talks.
So, quickly. What is ASO AC? You may hear us as NRO number council too, you actually elect us once in a while during these meetings. ASO, to go back a little historically maybe it will give some context to the newcomers too who are not familiar with these terms.
ASO in fact is address support organisation and it was a response to the creation of ICANN in 1998. So, we had celebrated RIPE community's birthday. It went 50 years, as you know yesterday, and you know by now, do the maths, RIPE community and require communities predates ICANN's formation. So, at the time RIR system was already functioning with three of them, three RIR communities, and regional policy making was already in place. With the creation of ICANN, there came this structural problem of how to connect these two things in one picture. And the answer was, the creation of ASO which started with a memorandum of understanding in 1999, being signed by the existing RIRs by them, three of them, RIPE NCC, ARIN and APNIC. Then Cure and MoU was updated with the creation of AFRINIC and LACNIC later on.
What is the scope. ASO AC, Address Council, is a function, and we review the global policy development process. This means any global policies that are regionally developed through the RIR systems needs to be connected to the ICANN and IANA where we ask the board on the last stage that everything was fine, all the processes was followed, and could you please implement this? So we do not get into the content of these policies at all. We are not policy making body. We just link that piece where ICANN board hears from us as a committee to look over all that, all the policies regionally was followed properly so that we stamp and then send it to them with the recommendation for implementation.
The other very definite task we perform is defining procedures for the selection of individual present I was to other ICANN bodies. We receive requests about this. And most importantly selecting ICANN board members, two of them. So, that's an important integral to represent the RIR communities on the board level of ICANN. We do that task of selection.
Our structure is pretty diverse in the sense of three representatives per region is seated. Two of them are often elected. And one is appointed by RIR board per region.
In RIPE's case, two years is the term for elected members. In some other areas this is different and one year term is noted for the, is in place currently for the appointed member.
For RIPE. So the last time around you elected these two people at the bottom of the list. Filiz Yilmaz, myself and Nurani and we have Wilfried as a board appointee on our Address Council right now. And Wilfried's term is up so if there are any, there is any interest, who would like to present themselves as a potential nomination for that seat again, please get ready because towards the end of this year there will be, the NRO will make an announcement and we'll then starting take nominations to fill that seat.
So, how we work is basically we work through scheduled monthly meetings. It's like a regular meeting with a full agenda. Often first Wednesdays of the month and we also have emergency meetings if we have an issue, and we also conduct face‑to‑face meetings at least in one ICANN RIR meeting. We are not required to travel to all the ICANN meetings because, frankly, our work is here. The RIR communities make the policies, numbering committees are made regionally, so AC members, it's often enough to engage with the ICANN community once a year.
However, recent work has been proving wrong in that direction. The agendas had been pretty full due to certain facts which I will talk about in a minute.
So what we have done in the most recent ICANN meeting. It was our annual face‑to‑face meeting, so all the ASO AC members were expected to be present in person. So we were all in Copenhagen, most of us at least, and we had an open session where we talked about our regional discussions, we provided informational session to the rest of the ICANN community, and we had a session with the ICANN board. In fact, if you have noticed this time, there is tremendous work being put there to bridge the gap and improvement of engagement. In this meeting, right, our board member, selected board member to the ICANN from the ASO, one of them they work really hard and one third of the ICANN board was attending and observing our meeting, so that's something important to mention here.
And then we had a session with the fellows. Fellows is a programme, ICANN programme to involve more newcomers in a meaningful way. They walked actually new members of the community throughout the week inviting representatives of the different diverse groups. So this is a diverse programme in fact ICANN does which our Task Force may want to have a look and learn from.
And what we are busy with. I said they have been busy. It sounds like oh okay, it's not the most interesting lightning talk, we could not see that in etc. But we think it is important that you know what your selected members and board appointed representative is busy with. Currently, there are changes going on of post‑translation in the ICANN area fora, and this resulted in various accountability Working Group streams. First they came up with, you know, the big part of it which was feeding towards the translation that is done now. And now, in fact, the working stream 2 is on diversity. And there is tremendous work going on there, and there is also RIPE NCC staff who is contributing as representatives to these formations. When I said appointing individuals to other ICANN ‑‑ other ICANN groups, it is not only us. We also choose from other people from the community. So that who are not attached to the ASO AC in fact. In that regard, accountability cross‑community Working Group representatives are, five of them, two of them are ASO AC Council members and the other three are from other parts of the RIR communities.
And there is work going on and we have to ‑‑ there is an impact of these changes in ICANN fora because our operating procedures as ICANN ASO AC has an impact. So now we have to sit down, following those changes, and to see how we can fit, how it will fit, do we want to fit is another question. So, there is more work coming on there.
Future ICANN meetings in 2017, there is one very quickly soon in June in South Africa. But what might be interesting to you is the next one in Abu Dhabi, which is just after RIPE 75, so, I believe there is a weekend in between and then the other one starts. So, two weeks in United Arab Emirates might be a future scheduling issue for some of you.
And that's it. Thank you.
If you have questions I may take. Otherwise, again, thanks for the time.
CHAIR: Thank you very much, Filiz, we sadly don't have any time for questions, but I'm sure you can find her...
(Applause)
In the hallway afterwards because we have to move on to our next lightning talk, which is ‑‑
AUDIENCE SPEAKER: Thank you very much. ICANN board member. Thank you very much, Filiz, it's a really comprehensive explanation about the ASO. I'd like to mention that this time for the RIPE meeting here in Budapest we had seven ICANN board members, two from the ASO appointed, but not immediate to them, we have the 7, ICANN board members here. Then it is actually ‑‑ we have, at the board, increased interest in other RIR process and then I am really proud and happy to have the 7, that's a very big number from the ICANN board members, that's just something I wanted to say. Thank you very much.
CHAIR: And now I'm sorry to be an American mangling his name. Here is a talk about another way of denial of service attack on all of our infrastructure.
EVGENY USKOV: Thank you. I am from Qrator Labs. In this talk we will discuss yet another denial of service vulnerability. It may seem for you quite simple and evident, but we found that there exists more than a million vulnerability of vulnerable hosts.
To begin, let's consider a typical router. It performs many different tasks, like building routing table, communicating with other devices via different protocols and of course an actual forwarding the packets. And there exists a well known abstraction that all these tasks can be divided into two planes with different properties, a control plane and a data plane.
So a control plane makes decisions about where traffic is sent. It runs various protocols like spanning 3 protocol, OSPF or BGP. The packets reach control plane if they are destined to a routing, or are locally originated from the router. But an important point here is that a control plane is usually handled by a central processing unit and, furthermore, since these operations are usually not performed for all packets, of course, there is no ‑‑ strict time constraint for processing the packets.
The data plane is also known as a forwarding plane and, as follows from its name, it does the actual forwarding.
So, an important difference between them is that data plane packets are handled on a hardware level, while in control plane, they are processed by a CPU, and so it is usually quite a bad idea to leave control plane open to the Internet because otherwise it can be used by attacker to reach the CPU.
A typical example of a way to access control plane is an open TCP port, and what the TCP port can be open on the network device, it of course supports network protocols. So we decided to take a BGP port and check how serious is the problem.
First, we performed a simple experiment to see if such an open port is really vulnerability and can it be used for denial of service. The victim of the experiment was a typical Cisco router, and on this device, we performed a SYN flood attack. The attack was intentionally made quite ‑‑ we ran 64 instances of HPing flood, it allowed us to generate quite a moderate amount of traffic. About 720 kilo packets per second and about half gigabits per second.
But, such an amount was actually enough for getting quite serious consequences. Here is the graph of the CPU load on the device. On the X axis the return and on the Y axis the HTP load. We can see for the first 45 minutes when there was no attack, the CPU was almost idle, but for the last 15 minutes during the attack it was almost full. But looking at this graph, you, of course, may ask, so what? What's wrong with ICP loads? There were much more serious consequences.
First of all, we observed a number of BGP session restarts. An example of a log file is shown. And we also observed several cases of OSPF reconvergence, which in particular resulted in unstable traces to the device. And the device was reloaded and it was hard to reach it, etc.
So, this experiment shows that an open port is a vulnerability which can be used for denial of service. And I want to highlight that this is not only about the denial of service of some single host, because this is a network device, and if the outage of such a device may cause much more serious consequences like the outage of the whole network.
So, the solution to this is, of course, quite simple. Just don't leave your control plane open to the Internet. Use access control or at least some private addresses or whatever else.
However, we found that, in many cases, a control plane of a router looks like this offering free hugs to everyone.
So now we present some statistics on vulnerable hosts. We collected data in the following way: We just scanned the entire IPv4 address space for open BGP ports and then performed some filtering, in particular we filtered hosts that answered on old ports and then we checked that what is behind this port. Actually looks like BGP. For example, it immediately closes the connection.
And here is the statistics. So there exists more than a million of vulnerable hosts. About one tenth of all prefixes is affected and about one third of autonomous systems. Moreover, about 5,000 of these autonomous systems are providers, so problems in them may cause problems in customers. And we also performed some checking of the behaviour of these open ports and found that most of them are just closing the connection. But there also exists some special cases. For example, about 60,000 of hosts sent a notification. And about 70,000 of hosts, before closing the connection, sent an open message. It really looks like offering free hugs.
So, the conclusion is that, despite this problem is known for, I guess, dozens of years, I mean the problem of open ports, it is still persistent, and once again, I highlight that in this case, it's not about the outage of hosts, it's about outage of network devices.
So, we developed a tool that anyone can check his autonomous system. This tool is free and available for everyone. You just need to register your autonomous systems, because it's required to prevent malicious use of the data. And if you have any feedback, please contribute it to us.
So, thank you very much. That's it. If you have any questions, I'd be glad to answer.
(Applause)
AUDIENCE SPEAKER: Is the two supporting both IPv4 and IPv6?
EVGENY USKOV: No. Unfortunately it's only for IPv4. But we are thinking about extending it to IPv6.
LESLIE CARR: I would personally be very curious if there are ASes that have IPv4 open and v6 closed or vice versa, it could be very interesting.
I also have another question. As far as, you know, has anyone exploited this in the wild yet?
EVGENY USKOV: Well, we don't know about such cases, and I hope there will be no such cases after this presentation.
AUDIENCE SPEAKER: Will, from AS26. I think I saw something like that being exploited because I got some DDoS on my BGP port that were open and that was trying to just like over load, try to connect and so on. And so I add some stuff like that in the wild but that was sometime ago. It's not so efficient because as soon as you put an ACL you are like safe‑ish, but it does exist, I saw that.
EVGENY USKOV: Thank you for the comment.
AUDIENCE SPEAKER: Christoph. Actually because I didn't hear it in the talk there, there is two things that you can do against it. This is either infrastructure ACLs on all your interface that is go somewhere, and second thing, control plane policy, just to add this.
EVGENY USKOV: Sorry, can you repeat the second point?
AUDIENCE SPEAKER: Control plane policy, which is putting like a policy in between the control plane and the data plane more or less. But, I mean, this is best practice, but obviously it's not what people are using in the network, otherwise we wouldn't see those open ports.
LESLIE CARR: Thank you very much.
(Applause)
And for our very last talk of RIPE 74, Paul is going to talk to us about challenges in digital archaeology.
PAUL THORNTON: So, morning everybody. I have been doing some presentations at UKNOF meetings mainly about Internet history. You know, 20 years ago in the dim and distant days, and actually this is a spin‑off from that, because there is actually some issues when you try and get information out of old routers, or computers and things that you really haven't used for sometime.
So, what's the challenge here? Well, electronics, they fail, you know, the components dry out. Bits fall off, you have bits that want to plug in, it's got the wrong port on the back, it just doesn't work any more, it's old, you can't replace it. And then when you do find this and you have got a tape drive or a disc drive or whatever, the media itself no longer works so you can't download that network diagram that you had, you can't get the router config, you can't power the router up, it's hard work.
So, the first problem. You have got to power these things. Well, power supplies are really really great but they have got components suspicious capacitors that dry out normally. If you are lucky, what happens when you turn this on is you get a puff of smoke and bank and it smells a bit bad. If you are unlucky what it does is it connects the mains power supply to the device you were trying to turn on and you get even more smoke and that's it for that device. If you are doing this sort of thing, make sure that your power supply is good first.
Your next problem. Batteries on PCBs. So, this was an old machine of mine that I was working on probably in the late nineties, and I opened it up and this is a really common problem. You get these really furry batteries and you have got to desolder them and they do a lot of damage to the circuitry that they have leaked around. Luckily, normally vinegar gets this off and you can tidy this up. In the olden days, you had components there that you could solder yourself. Trying to do that with something from now in 20 years' time would be a near impossibility.
There are other problems that you really don't expect. Mice can move in and leave you little presents. And going back to Daniel's point of the spirit of brutal honesty, getting mouse droppings off a circuit board is really hard work and something you do not want to have to do. We did actually make that work eventually.
And you then discover in the same machine when you have taken these old cards out that the mice had to eat something to make those droppings so they have been chewing on components as well, so you may have to do some more repair work.
Then you finally got your computer or router actually working, you want to plug it into a network and you go, oh, crikey, so I need to find one of these things. Now, I'm showing my age and people who are chuckling are probably showing theirs that, they actually know what this is. So, at least this one doesn't have a 10 base 2 connector on the side the but on the back of some of these things they had AUI connecters which was a 15‑way D‑connector and you had to plug a transceiver in before you could plug it into your switch. You plug it in and then you plug it into a switch and you go, oh, that port only does 100 meg or a gig, I need to dig out some old switch or hub that I can use to actually bridge the gap.
So what about other things? Let's say you have got some data on a disc or a tape. Well, that's okay. Because it's got a SCSI interface, and, you know, this has actually been one of the serious points. SCSI has been a good standard because way back when in the 1990s, early 90, to a mourn SCSI card today it will probably still talk to that device. It will go slowly, but wait that's a PCI bus, I don't think I have a PC with a PCI bus in it any more. You have to keep lots of additional rather which is no problem for me because I never throw anything away. It means that when I have needed to get some of this stuff, at least there's been you know, equipment that could help me.
Then you find some old media. These are ‑‑ so, these are DDS2 tapes and a jazz drive. Well, DDS, that's still a standard. But there is backward compatibility issues, you can't put these in a modern drive. It makes noises. And a jazz drive... well, I could have got one on eBay I thought the probability of this thing reading, it's not even worth getting one.
So, you eventually find a dak drive that will take this tape. You go this is great I'll try and find the data on it. No, nothing comes back from it. What's concerning there is that it took 48 seconds to realise it couldn't read this tape and was making some really nasty noises. So I'm hoping that the tape is not completely shredded now.
So, on a serious note. If you are doing things now that might ‑‑ you might think this is going to be the next great thing. Someone in 10, 20 years time might want to actually come back and look at some of the really early bits that help make what you're building. So, how do we keep this so there is a trail for the future?
Well, the one thing that never let me down was a 1 gig IBM SCSI hard drive that I bought in 1994, it's still going strong now. Again, it's SCSI, but it works with a modern interface and hard drives, the ones that I have spun up, have managed to give me data back. Hard drives are definitely better than types for this.
If you are going to store something in a file system, use the lowest common denominator. Fat 32 is not sexy, it's old, but pretty much every operating system can read it.
Well‑known formats like tar or zip. That's great too. I was talking to someone earlier this week who had made an off‑line bitcoin wallet, he was going to store it on an encrypted zip and put it on a USB drive. He wanted to cash in some bitcoin, unfortunately he couldn't remember what passwords he was using six years ago, so he had to make a list of all of these passwords he used. You don't get ten goes and capital wipe your device. You have got a problem there with sort of combining security with accessibility later.
Avoid propriety technology, the jazz drive. Make image files. If you have got a disc or router or the Flash, use that DD, store it as a binary image and you can come back to it.
What I'm doing now is you move it forward to the next raid cluster or whatever you are doing. Keep it on live spinning media because that's less likely to let you down in the future.
If I questions? You should always have happy monkeys.
(Applause)
AUDIENCE SPEAKER: Benedikt Stockebrand. You can still get main boards with PCI or even ISA slots for ridiculous money admittedly but there are some people who are some speciality ISA‑only cards. There is, if you need to get the data off you can get that, and other than that, why didn't you mention the cockroaches? Mice are harmless.
PAUL THORNTON: They were good. The mice had moved out. It's fine. They just left their traces behind afterwards. And I have seen those ‑‑ it's things like big dot matrix printers on eBay with more than they cost new because of these really old systems that need them. He can get this stuff but it's not cheap.
BENEDIKT STOCKEBRAND: You can really get pretty much brand new main boards with these old slots, because people actually need them for some stuff. It's worth it for some situations.
AUDIENCE SPEAKER: I have a colleague that is skipping already every above in the last 30 years, feel free to make a trip to France and take everything, you'll be very welcome.
PAUL THORNTON: I think my colleagues are already trying to make me throw some of this stuff in the office would not like you for that.
CHAIR: Again, Paul, thank you for the presentation.
(Applause)
So, I made a mistake, now we are closing the Plenary actually, so the LTs of course are part of the Plenary programme. I want to hand over the mike actually to the PC Chair, and ‑‑ sorry, the PC Chair ‑‑ the RIPE Chair, and ‑‑ Razvan, please go ahead. It's part of the RIPE Chair session.
RASVAN OPREA: Good morning everyone. I see a lot of people still being here, I am very happy. I'm sure that this technical report was what kept you from leaving so far, so it's exciting.
Let's start with the connectivity. We had ACE Telecom being our connectivity sponsor. You are going to see the stat slides a bit down the road, that their service was impeccable. We had 1 gigabit fibre and 3 megabit radio backup which we didn't have to use at all.
What you see in the speed test screen shot is simply something that at random this morning we went to the Terminal Room and we plugged the wire into a laptop this is what we got. It's completely random. It doesn't say anything about separating the link or about getting I don't know how many gigabits per second. It's just something that shows that if you wanted, you could have had your files downloaded pretty quickly.
This is the equipment we used. You can see on the right‑hand side, the hotel network being patched with our equipment in a very neat way.
On the left‑hand side you see our new routers, we have moved from Junipers to Ubiquiti, where they are still new, they are still learning things about them. Actually, at the beginning of the meeting on Monday, we had a strange issue with duplicate packets on the network. So after investigating and investigating, we did what any experienced network engineer would do, we just unplugged the second router and everything was perfect. That explains what you see in the picture, just one router is being plugged in and the second we used as a cold stand by.
On the wifi, this was an interesting meeting. We have the old Aerohives for some years now and they came problematic sometimes, either look‑ups or general problems with them, so we're looking to replace them. So, we bought ten Ubiquitis that we installed in the foyer area, so we are looking to you to give us some feedback on whether the foyer area connectivity was any different than the one in the meeting rooms. We have the IT support desk just outside of this room. A lot of people came over there with questions, some of them were technical in nature. Thank you for that. And most people that came with wifi issues seemed to have issues more in this rooms. Yes, it's high density, but in the barista it's also very, very high density at times, and this problem seemed to have solved while they moved away from the Aerohive area towards the Ubiquiti one.
They are not without problems. The new access points, they have this old problem with radar detection on the 5 gigahertz band. The symptoms that we have seen were that once they HOP channels, the people that were connected to them remain connected, but they don't accept any new connections. So, we tried to troubleshoot the issue. We could not. We contacted the 24 hour hot‑line support, and their advice was to use non‑DFS channels, so thank you very much, Sherlock. In the end, this is what we did but we were not very happy with the solution because we would like to use the entire spectrum of course so we'll see how we can do this by the next meeting.
In the Open Source Working Group, we had Andrei explaining how hard it is to manufacture devices, physical devices. Well, we did here custom‑made mounting brackets, and it just went fine and quick. You can see almost patented version on the left side for the Ubiquitis, and the Aerohives, which runs so hot that they can actually leave marks on the carpet, so basically we need to put them on to something and that's not because of so many people, it's because that's how they are, so what you see over there underneath them is just super bowls turned upside down, and they are in a safe position underneath the chairs.
Let's look at some stats.
What you see here is the network traffic. It was still very reasonable. I believe the peak was around 150 megabits per second. It's generally within the trends that we see. You see the download and the upload downwards. It's very much in sync with what you have seen the previous meetings.
The DHCP leases. We have close to 800 leases being offered. Again, this is something that grows slowly meeting over meeting and you can actually see the daytime you see the first, the first growth, the slope is going lower during the night because it's a Monday, a lot of people saw each other again and stayed around. And then there is this strange thing on Wednesday night, we haven't figured this one out yet.
So, we used the RIPE Atlas to show us the ping measurement to an Altas Anchor in Amsterdam for both v4 and v6. This graph is not normalised because when you look at it you go, wow, look at those peaks. But if you are curious to download the slides and look at the numbers over there, we're talking about 1 millisecond between the highest and the lowest. It's basically a flat line.
These are ping measurements to the K‑root node. You see it's flat zero almost because there is an instance that we manage here in Budapest. And down you are going to see an I‑root node with v4 on the top and with v6 faster, lower latency on the bottom.
Wifi statistics. How many clients do you have? This meeting we had over 1,000. It's again in line slowly growing meeting by meeting. Mostly 5 gigahertz client as compared to 2.4 gigahertz one. So I think this is something good.
And I just want to show you the comparison between the previous meeting and this one. Hopefully, you can read that the blue, the RIPE MTG has been used more this meeting. Why was that? Because if you are going to look at what SIDs you can see, we changed the name again, we just tend to do that. It's going to be the RIPE MTG is the one that is stable that we actually encourage people to use and then the 2.4 we are no longer using one single SID, we decided we are going to change it every meeting so this one you saw 2.4‑74. Next time, it's going to be 75 and so on. So, this prevents your devices from remembering the 2.4 network from last time and just reconnecting to it automatically and just by doing this, we have another 10, 15% increase in associations with the 5 gigahertz network.
On the NAT 64 network it's pretty much the same adoption rate. Let's say that 14% of you have used the NAT 64 either on the 5 gigahertz or 2.4 gigahertz, so that's quite a nice thing.
A huge amount of people are actually trying to work and make this meeting successful. We have webcast and presentation system. It's this way of pre‑loading presentations while the presenter has not finished speaking yet, like what they are doing now, for instance, for the next speaker. You have here our photogenic Manno and Christian but there are so many other people that are working from the RIPE NCC to do the webcast and the presentations.
Who are these?
So thank you very much, Mary, thank you very much, Aoife, thank you very much. Ronan is in the other room as well. Like always, fantastic job.
Web services team. They take care of the RIPE 64 website. They do everything that's web‑related. Converting presentations, rating system, uploading the presentations in line with your editing and also a new system, submission system for the Programme Committee. So everything...
(Applause)
And finally, not least, is the operations team. With the addition of Csaba, with the addition of Csaba who came specifically for this meeting to help us out. Thank you very much.
(Applause)
If you have any questions or comments?
AUDIENCE SPEAKER: Hi Raz, Elvis Velea. I'd like to ask you to go back a couple of slides to the one showing the graphs. The one with the traffic. Actually ‑‑ no, the leases. Because I was very curious to actually see how many leases we had on Wednesday evening, because the hotel will ask me a few questions about that. We probably had a bit more people than we thought we'd initially have during the very important BoF that is not to be mentioned.
RASVAN OPREA: About 145 to be correct...
AUDIENCE SPEAKER: Thank you very much for that. Indeed, one comment about the wifi. It has been better on the outside than in the room. That was my impression. We have had a few chats, there have been some weird things happening in the room with my iPad, I could not connect to various of the access points, or maybe the ops team were blocking my access five minutes here, five minutes there, just to play tricks with me, I don't know exactly.
RASVAN OPREA: It's rate‑limiting put on if you ask too many questions, yes.
AUDIENCE SPEAKER: Martin Levy from CloudFlare. I want to make a plea and maybe people will agree with me in the room or not, but I think it's time to dump the NAT 64 network. Just move that over to RIPE meeting and run a SYN in the 64 environment. I am ready to take that plunge, I don't know who else is. But if we can't handle it how do you expect the rest of the world to handle it?
RASVAN OPREA: Thank you very much. If my memory serves me correctly, it was a recommendation from the IPv6 Working Group to do this. So, if you take it to the mailing list and there is a consensus of this, yeah, sure.
MARTIN LEVY: My point would be that's a great place to discuss it, it's a great place to discuss it, but I want to make it a little bit more mainstream and I'll pay a pitch for this, 500 people at the Facebook at scale event two weeks ago and they just ran a NAT 64 network, full stop. No option. And there wasn't anybody complaining in the room. So... sometimes you just got to jump in the cold end of the swimming will pool.
RASVAN OPREA: Thank you.
AUDIENCE SPEAKER: Philip. First of all, thanks for this event and the network and the video streaming and so on. We know especially how hard it is to provide wifi for the mass, it is quite hard, and high density it much harder. We are happy if you need some help for the next event, we offer some support to share our knowledge.
RASVAN OPREA: Thank you very much.
AUDIENCE SPEAKER: And at least, don't buy Ubiquiti wifi for high density, trust me, we learned in the painful way.
RASVAN OPREA: Let's have a chat.
AUDIENCE SPEAKER: I am happy if we could help and assist you. Raz, we are open to any suggestions. We haven't decided on going on them but they are a strong, they are under strong consideration, so we'd love to hear it if you can help us.
(Applause)
HANS PETTER HOLEN: So. All good things come to an end. We have been here for a week now and it's the closing of RIPE 74.
We had 629 attendees checked in. The last attempt to find two more people on the street so that we could set a record from the last meeting failed. So we're one down from the last meeting. But it's still a healthy large number which is very good.
Looking at the number of newcomers, we have 24% newcomers, which is very good. There was some questions about retention here, so you see that's 76% of you have actually come back, which I think is a very good sign.
And we would like to know what you think of this meeting. So, rather than all of you running to the microphones now and tell us, please answer the survey. We draw a prize, you can win one of two Amazon vouchers and there is also a third prize donated by a community member, which is a voucher for a go fund me. Instead of spending money on yourself you find a good project on Go Fund Me and use this voucher to fund that project, which I think is actually an excellent idea, we need to collectively and individually fund good projects in the Internet community.
So, you have all clicked on this URL now so that you are ready to do the feedback, right?
Attendees are from more than 60 countries and I got a list here of the biggest countries in our regions. Any guess? US is number one, Germany close by, and the Netherlands following that. So we have to do something about recruiting from some of the larger geographical countries in this region that was commented here and it's going to be very interesting to see what proposals we get for meeting locations next time we call for meetings, because we decide the meeting locations partly based on a strategy that we want to cover the whole region, but we do depend on getting proposals from local hosts and we do depend on having infrastructure like a hotel with venues that are big enough.
Types of organisations.
Not all of you are commercial. Actually, less than half of it. We have a healthy flow of other associations, education and even some governments taking part in this, which I think is very good.
Then this meeting would not happen unless we have Working Group Chairs, so I want to give all the Working Group Chairs a big thank you and please give them an applause.
(Applause)
Some of these Working Groups do actual work and have policy process and so on. Some of them are more like special interest groups and there is also a group that met in an evening this week and there is a proposal to form a new Working Group...
ANNA WILSON: Hello. Thank you very much. There was an Internet ‑‑ Internet of things BoF on Tuesday, it was super well attended. There was a lot of energy in the room and without talking you through the whole thing the conclusion of the BoF was we probably need a working group, so a few people stayed behind to start the process of putting a charter together or at least some rules for a charter that we could discuss. That's what's on the slide in front of you and that went out to the IOT discussion list [at] ripe [dot] net, if anyone wants to comment further on that. It's not the charter, we think it's a good place to start. And what we'd like to do here, if the Chair will allow us, is seek approval from the Plenary to go ahead, start the process of forming a working group, with a view to actually discussing a charter and approving it in the next meeting in Dubai. And also perhaps appointing co‑chairs in Dubai, at which point, of course, we'd come back to the Plenary and ask for formal approval of that.
And that's our plan. I know we're short for time so I won't get into why I think it's necessary to have a working group as opposed to a BoF, but I'm happy to if there are questions about that.
HANS PETTER HOLEN: I think this is an excellent initiative. It's very high on the agenda also outside this community, what's happening in this space of Internet of things and securing those. And the proposal to move ahead in the direction of creating a working group is very good. We will have a look at the meeting plan the next time and make sure there is an agenda slot for this initiative and based on what's happening in that slot, we may or may not move forward to create a working group. So thank you very much for the good work.
(Applause)
So, some words about the core values of this community. We live by open, transparent and inclusive. We had a session this morning directed towards open and inclusive with looking at how we can make this community even more open and inclusive. We heard several good ideas on how to do that and I am also very pleased to hear some people saying that yes, I have really experienced that we are open and inclusive. I was thinking back to my first RIPE meeting, and I must thank Wilfried for standing here because he included me in the Database Working Group by making me scribe in the very first meeting, so, he put me to work so I felt really valued back then. We were not quite as many in the room as today, but I think this is a lesson for us to still think about how do we receive each other and make sure that the newcomers, not only feel welcome, but are also put to work and contribute to the community.
So moving ahead with that initiative, it's going to be interesting to see what are the suggestions, do we pick up, how do we take this into the future? Do we need a task force for this and how do we do that? That's something that we will discuss up until the next meeting.
We are also working on transparency. There is a task force working on looking at how are we accountable. And there was a presentation here on Tuesday morning I think by Filiz, and that Task Force is now at the stage where there is a draft scope out there. There was some discussion to that. And what they really want to do now is to sort of finalise the scope. So that they can set a deadline and I heard one of the chairs said that they had a tentative deadline to produce something before the next meeting and I think having a document from this Task Force at the next meeting and have a discussion then would be an excellent progress in this area.
And then, the content of this week would not have been what it is without the Programme Committee. So in order to thank you we have some gifts for you, so if the Programme Committee could come up here, Martina has some gifts to hand out.
(Applause)
So, now you can all do the bow that you rehearsed!
So thank you very much and a special thanks to Shane, who ran for the election but did not make it to the next PC. And a big welcome to Franziska and to Benno who will stay on for yet another term. So thank you very much.
And it's already been said, thank you very much to the stenographers for learning ‑‑ acronyms, slang and names, and we're looking forward to seeing you at future meetings as well.
RIPE NCC was 25 years. So, happy birthday RIPE NCC. There was a quiz. And nine of you actually got perfect scores on this quiz which is impressive. Axel drew the name from the hat from these nine quizzes at the dinner yesterday, and Piotr, are you here? You were the lucky winner, and here is your prize, if you want to come up and collect it.
(Applause)
And then we usually have prizes for the first two registrations. The first one this time was one of the Working Group Chairs, Gert.
And the second one is Paul Burrows. So in order to get the prize you have to be in the Closing Plenary, so unless Paul Burrows is here, I'll go down to the next one on the list, which is Alex Saroyan.
HANS PETTER HOLEN: Nick Hilliard. Well, without our host, Janos, this meeting would not have happened here, so, Janos, please come up and receive a big thank‑you to you too.
(Applause)
It says here the Council of Hungarian Internet providers and Budapest Internet exchange, he has a big hand in those two organisations, so thank you, Janos.
And then, of course, thanks to our sponsors. Without sponsors you wouldn't get coffee, you wouldn't get all the nice things you have in lunch and so on. So, once again, thank you to the sponsors.
(Applause)
And then there is only one thing left before you can pack your bags and go home. And that is where you are going next. That is to Dubai in October between the 22nd and the 26th, and that's a meeting from a Sunday to a Thursday to align with the work week in Dubai, so it's not Monday to Friday, but Sunday to Thursday. And that's all. So thank you very much for this time. And safe trip home.
*******************************
