IPv6 Working Group
10 May 2017
2 p.m.
JEN LINKOVA: All right... hello everyone. Thanks for being here, I know it's a hard choice. I also would like to attend MAT session.
So this is IPv6 Working Group. Let's grab your seat and fasten your seat belts and put your phone in the flight mode.
We have a pretty busy agenda today so I'm going to be quick and leave the stage for presenters.
Does anyone have any suggestions, comments, objections to the agenda of the last minutes? No. Good. Then, let me thank our stenographers and Jabber scribes for being here and helping us talking about IPv6.
Only one administrative matter I would like to discuss. I know I did it a little bit later than I should have, but I did send meeting minutes from the last RIPE meeting to the list. I am not going to ask who has read them, but any objections we approve them?
3... 2... 1... sold!
And last reminder, please rate the talks because otherwise we, Working Group Chairs, have no idea how to build the agenda for the next meeting. So, if you do not rate the talks, do not complain about content of those sessions please. I mean you could, but ‑‑ rate the talks.
Our first presenter today is Enno.
ENNO REY: Good afternoon everybody. I am going to present the results from some lab testing on source address selection that two students preferred last year with some supervision and guidance from myself.
A very quick note on who I am. Most of you know me probably from earlier RIPE meetings or from the Plenary presentation I gave yesterday. I do a lot of IPv6 stuff, that should be sufficient to notice at this moment. Then there was one of the two students, he does a lot of IPv6 stuff as well, and some of you might have stumbled across his name when you did, or had to perform some patching of Cisco devices in March. He is the guy who found this stuff. So he is involved in a lot of security research and in general, lab activities at ER networks W. He is not a student any longer by the way. He is a full‑time employee since the beginning of the year.
That said, what am I going to talk about? As I mentioned, the research question was how well does IPv6 source address selection as of mainly RFC 6724 perform say in different settings and with different operating systems?
We did this testing last year in July and August. So, I'm not entirely sure if the results would apply today. As well as. I assume they do, but this is not, say, it hasn't been tested four weeks ago. We plan to renew the testing at some point, but I think the results are still valid, and maybe it's not so much about the specific results, but more about conclusions we have derive from the things we saw in the lab. The problems we solved were the once I mentioned yesterday, once the system has several addresses, there is a decision problem, which one of those to use for an out bound connection. And this is even more important or might become even more interesting as the destination might have several addresses as well.
What can already been derived from this picture is there might be some interaction with DNS. We can expect that the majority of connections won't be established as for a destination address with the address, or the initial point of reference, but based on a DNS query. So DNS might play a role and we will see that this actually interacts in some way with the way the source address might be chosen.
In general, we should keep in mind that source address selection is only one element in say more or less complex setting, or in a chain of decisions. At least in a dual stack setting we can assume that systems interest have IPv4 and IPv6 at hand so a decision has to be taken does the system use for, to reach a specific destination, assuming that is is dual stack as well, is IPv4 going to be used or IPv6 going to be used. Then things like the so‑called prefix policies might come into play like prefer a native IPv6 or a tunnelled IPv6 connection. Let's assume for the moment that the amount of tunnelled IPv6 is decreasing, which is a good thing in any case.
At some point, or in this decision, say, setting, IP eyeballs may come into play at some point and Windows does things a bit different than happy eyeballs, they run the ICS eye player roll. This might interact with the establishment of a connection and as I mentioned the DNS plays a role too. So just looking at source address selection might mean that one overlooks ‑‑ one has to keep in mind there are some more stuff that has an influence.
What we did is quite simple. We tried to focus on the RFC 2764. We set up a simple lab which I'm going to describe in a second, and we looked at different scenarios.
That said, what is the expected outcome? For source address selection there is mainly two RFCs, 3486 which is obsoleted by 6724. To our pleasant surprise, all operating systems we looked at followed 6724, which is a good thing. There is some, Jen has a draft which suggests some addition to a 6724. Do you want to very quickly comment what ‑‑ there is an additional rule ‑‑
JEN LINKOVA: Okay. So the problem is currently if there is no other tie breaker and we went through all the rules if all addresses are equal, a host prefers the longest prefix matched with destination. However, there are scenarios where the host has two different prefixes received at a different time. One is old, one is new, and for example a router was rebooted and host did not know its network had changed so in this case a suggestion is one of the last rule is to use the most recent address configured using SLAAC. So basically to cover scenarios of DHCP prefix delegation and other times when the host did not detect the network change.
ENNO REY: I think that's a useful digs. 6724 might be updated at some point by another RFC, just to let you know.
There is, in the context of address selection, there is one other RFC that should be mentioned that is 7078 which describes the possibility that a DHCP server actually distributes say has an information bit which approach to follow. I'm not aware of any implementations of this. If any of you is, I would be interested to hear how well it works or how well this is implemented. But for the moment, let's assume 6724 is the main spec we should look at.
What has to be considered before we look at the rules of 6724 actually, say prescribes, what has to be kept in mind is applications, or 6724 describes say default behaviour of a stack. Applications can at all times override the behaviour of the operating system of the stack.
The approach of 6724 mainly is, once there is several candidates to use, several address candidates to use for an out bound connection, put them on a list, apply certain algorithm which sorts these addresses according to specific rules which I'm going to mention in a second, and once this has happened, the best one is handed back to another process in the stack and is going to be used for the outbound connection. These rules are these nine rules, they are ‑‑ actually it seems it's eight rules, but there is for whatever reason, there is a rule 5.5, which is a quite interesting one for certain scenarios. So, it's nine rules. I'm not going to cover them in detail, not least for time constraints. The thing is there is some very simple rules say of destination and source, or that's the first one are the same, then use this ‑‑ if the source address is the same as the destination address, use the source address. Which is pretty obvious.
From these ‑‑ I would say from these rules, the most interesting ones might be rule 2: Prefer say, keep the scope in mind, so, a scope from the, the scope that can be used in Multicast addresses but, say, prefer or keep in mind if it's a link local connection versus a global one. Rule number 3 is pretty obvious. Or makes sense. Don't use once an address is in the lifetime, is in a depricated state where it can still be used for established connections but don't use it for new connections. I would say the most interesting ones might be 5.5, keep in mind, which ‑‑ if there is a relationship between the next hop and an address you use, don't use an address which wasn't, say ‑‑ which doesn't have a next hop. And obviously rule number 7, prefer temporary addresses over non temporary public addresses. And rule number 8, many people know rule number 8 like longest matching prefix. Here it should be kept in mind, rule number 8 is kind of tie breaker rule. It's used as the last resort, and it doesn't have to be used as of my understanding of the RFC.
And there is a nice ‑‑ I'm not going to cover this ‑‑ sorry, I'll skip this. There is a nice diagram from a book on this decision tree. It's too small to discuss here in detail anyway. I put it on the slide, I can't go back for whatever reason. But the thing is, these rules are not used in a weighted manner like calculate something, looking at edge rules but in an ordered manner. Check the first one. If there is a decision, go with this. Check the second one, check the third one and finally maybe reach rule number 8 so you can look at the longest matching prefix.
A very simple lab setup. Two segments. One web server which was reachable and displayed the address which was used for the connection. And two main test scenarios, half the clients in the same web server half the clients in a remote segment which is probably the most realistic use case, and equip the clients with different types of addresses and have a look at what the client actually choses.
I will skip some of the details. There is a paper we put out with the results and the details which type of configuration to use in the specific test cases on the Cisco router sending the router advertisements.
These are the operating systems we looked at. And now let's come to the most, say, interesting thing which is probably the results.
I will skip the exact test scenarios, they are all described in detail, together with the relevant configuration on the router.
The results:
There is some interesting things we could observe. In the first test case, the first test case was just use link local. Don't have a global address. There were some differences between the operating systems, some connected, some didn't connect at all. Some could get a DNS server, that's the result of address server. Others couldn't. At the time Windows didn't support RFC ‑‑ or now 8106, so rDNS S option in router advertisements. In the interim for those of you who haven't noticed, they do. The latest Windows, the update which was released I think three weeks ago, Microsoft has RD networks as support as of today, which is a very good thing. I might add it would be nice if Google had support for DHCP 6. But there is movement in the space, which is good.
Let me skip this in the interest of time and look at the results pretty much immediately.
There is some interesting things to see here. The first thing one may note is in general, and this is a good thing and this was a very positive surprise to us, in general most operating systems behave consistently. There is some small exceptions here and there, but overall, there is good consistent implementation of RFC 6724. The interesting deviations I would say are the following ones.
Again all this is described in the paper in more detail.
What is interesting is, first of all, there is a deviation of, say Mega RSX and iOS, for whatever reason, we couldn't figure out why. To be honest we have no idea. They did not always use, when they used the global address, they did not always use the temporary address but the public one for out bound connections. Again I have no idea why, this is a clear, what I think, say deviation from both RFC 6724 and from ‑‑ I mean, why have privacy addresses, or temporary address ifs they are not used? But again, I don't really know why. The second thing, and that one is quite interesting as far as the interaction with other mechanisms. You might note that for the test case 5, the test case 5 actually was ‑‑ have a ULA address and GULA address, in parallel and have the ‑‑ in DNS, have two AAAA records for the web server, the web server also had ULA and a global address, hand out two addresses ‑ And the thing here is; the Windows opperating systems do not behave the same and we think the main reason for this one ‑‑ but the thing is, apparently Windows, when I get a DNS response, with two AAAA records, there is some Windows derivatives, so to say, which, in the protim in the round robin, there is others which look at the record and then identify oh wait, I might have two options, but I always chose the global option and there is described behaviour, there is a registry key which adjusts this behaviour and the default state of this key has changed between Windows versions.
This brings me to my conclusions. As I already mentioned, overall, there was a high consistency as for following RFC 6724 rules. We had described like expected behaviour in advance and we have we were pleasantly surprised to see in the majority of cases expected behaviour was met the when there was differences, those differences, from our understanding and from our interpretation, arose not by, say, different implementation of RFC 6724, but by other parts of the operating system. Namely how to handle DNS responses. So, 6724 didn't come into play. It was still interesting to see there is differences, but those were not related to source address selection but taken by other mechanisms which then again, and this is my last statement here, emphasises, underlines that just looking at 6724 a source address selection might not give the full picture. One has to sustained what does a system, which components come into play for taking a decision. And keep in mind it's several components, it's not just source address selection.
Thank you for listening. I am happy to get questions.
(Applause)
JEN LINKOVA: Questions?
ENNO REY: All this, as I mentioned, there is a paper published which has much more details than I gave in the last 20 minutes.
AUDIENCE SPEAKER: Gert Döring. Actual selection is something that I find not that much interesting, what I find much more interesting is source address fail‑over, like you have multiple global addresses available, you try one it's not working because something in the path is broken. But one of the others would be working if you would just use it. Are you aware of any work at IETF to actually tackle that? Because right now, the 6724 says pick one, use that. If that doesn't work use a different destination address but nobody ever mentions you could use a different source address.
ENNO REY: I would like to hand it over to you as you probably know more what's going on.
JEN LINKOVA: I think there is something. I don't want to miss inform you, mislead you, but we can talk off line. I might point to you people who might have some ideas.
ENNO REY: I think I know that someone on the mailing list from baker or so on like applying happy eyeballs, similar thing, to v6 only scenarios. So, choose amongst them ‑‑ once you notice the exact scenario you that described Gert, once you notice one doesn't work, pick another one. I think there is something going on, but I can't give you the exact name of the draft or something, but I can probably give it later to you.
AUDIENCE SPEAKER: Benedikt stock brand. To answer this question, I'm not a hundred percent up to date what's going on at the IETF. But because the statelessness of this, of IP as such, if you don't think TCP but UDP, you try to send out a packet, you have to pick one source address and you don't care about if it works or not because you can't tell if you just send out a single UDP data gramme in an IP packet. So there is very little you can actually do to try multiple addresses. You could do this in layer 4 if we're talking TCP where you get a synac back, but not in layer 3 alone so we'd have some solve to have notification thing or whatever and then things do get decidedly complex
GERT DÖRING: I think having something for TCP is already good enough, because it's ‑‑ well it's better than what we have today which is nothing whatsoever. IETF is preaching you should do multi‑homing, by having multiple global addresses of same scope and then the declaration is all defined when it's complicated, let's go fishing. So this is why I'm interested in whether somebody actually picked up the slack. I know that people are aware of this, so I was curious whether actual work was being done so I'm waiting for your off line reply. Thanks.
AUDIENCE SPEAKER: Marcus Keane, Microsoft. There is some work going on in the interior on the notion of provisioning domains which has multiple prefixes on the same segment. But as far as I know, that solves a problem of choosing which source to use for which address space, but I think the idea of how you react to a failure of withdrawing that provisioning domain, I think that work is unsolved.
ENNO REY: Is that the draft that was initially drafted by could have an or something?
JEN LINKOVA: Networks it's related but it's an inter provision domain. I have two very, very quick questions. So, first, when ‑‑ did you look actually back to the happy eyeballs for addresses, when you say that machines were choosing addresses consistently, did you actually try both at the same time and just connection was established over one?
ENNO REY: No.
JEN LINKOVA: You didn't even try.
ENNO REY: Yes.
JEN LINKOVA: Have you looked at rule 35 an in case of two routers advertised two different prefixes and host user?
ENNO REY: No we haven't.
JEN LINKOVA: I'd love to see that.
ENNO REY: I know that 5.5 is particularly interesting, not least for the context of the other draft that you mentioned, about but we haven't looked at it. The only occasion I stumbled across 5.5 in the past is where it was my only explanation when you have a DHCP assigned which doesn't have a route interface in the same sub‑net ‑‑ but clients could communicate over the DHCP addresses but don't, they pick a SLAAC source address, and my explanation was this must be 5.5. But I have not stumbled across that, or not tests other scenarios with 5.5.
JEN LINKOVA: I might confuse our audience completely. I was talking about scenario with when you have two routers in your network and one is sending you in prefix for configuration and another router is sending you another prefix for configuration and a host is expected to select as a prefix which was advertised by next hop. So if I go to router one I go into use address advertised by router one? SLAAC. I am looking for to updates on your research because I am very interested. Thank you very much for presenting it.
(Applause)
And we are going to continue talking about IPv6 addresses.
FRISO FEENSTRA: Sorry, mints today. Aid big bowel left over last night. I don't know where that went. Registration desk. If you are anxious for another mint, please go to the registration desk and I saw a bowl of some very old IPv4 mints over there, so, but don't really take those.
So a little bit more about Rabobank. We're a corporate bank, third largest in the Netherlands. And I'm going to talk about the way we set up our IP number plan but to really understand that you have to know something about the topology we have, because IP number plans should, in a sense, reflect the requirements from that topology.
Now, our topology in the Netherlands, Rabobank in general, we have two main large divisions. One division is Rabobank for the Netherlands. We're a Dutch bank so that's also the biggest division. And we have Rabobank international, which is covering all the other countries. And that's Rabobank international. So the Netherlands, we have consolidated all our data centres into two and actually it's not really two data centres, it's more you can consider it as one twin data centre, both locations are relatively close to each other.
We have 800 standard office locations in all small villages, towns in the Netherlands. Most locations still have a Rabobank office. And on top of that, we have 300 unmanaged, what we call unmanaged locations, and those are normally our ATMs, because an ATM for us it is a network location because there are multiple network devices at such a location.
International, you see the details, and it's expanding. Fortunately, that part of the Rabobank is still expanding.
For this, of course we have IP space. Initially, we got /16 in IPv4. We still use the /16 partially internal because we have got an internal connection to what we call our allied parts of the company. Those are parts of the company which are owned by Rabobank, but they are still independent by themselves to we have a type of direct connection to them. But since they are more or less independent, at least network wise, they have their own IP space. But in communication with them, we use public IP space. And part of that /16 is also external facing IP space, that's the IP space we advertise on the Internet and that's how you reach Rabobank.
For IPv6, we got a big block, we were very happy with that, a /29. Our AS numbers 8211 for the Netherlands, my department. And we have got other AS numbers in other countries as well. That's the international part of our company.
First thing we did was we at least more or less subdivided it. IT I, the part doing the international networks, they got the first /32, and then we said well let's keep some space that are not allocated, and then somewhere in the middle, the CC 4, let's use that as Rabobank Netherlands. So actually, within the Netherlands we are using the /32, which means that the other blocks we have not allocated yet. And I'm quite happy with that because something like 10, 15 years ago, when the IPv4 number plan was reused, internally we use private addresses, on IPv4, and somehow we managed to use up all IPv4 private addresses, all blocks. Now that in itself is not that bad. But you have got no way of migrating from one number plan to another number plan because there is no next available part of the number plan.
Now with these current allocation, we can use /32s, and I anticipate that that will be enough for about 10 years, and then the next ten years we can use the next 32. The next 32 and the next 32. Enough that in 40 years, hopefully the first 32 will have lifecycled out and you can start using the first /32 again. And then we are 50 years ahead and I hope I won't be around by that time.
The importance of IP number plan specifically with IPv6 with those long addresses, is to keep it as easy, as simple enough so that everybody understands. I think that should be the basic of all communications; keep it really as simple and straightforward and idiot proof as possible.
The other thing is keep is summarisable. You don't want a lot of prefixes on your network. Recently, we have done some testing with new high speed HP equipment where we were feeding the, just for testing, our router instances big IPv4 and IPv6 tables and we were measuring convergence time and one of the things we noticed, that convergence times do have a linear dependency on the size of the routing table. So, the bigger the routing table, the higher your convergence time. And then that convergence time, with big routing tables, can go up to seconds. Now maybe on the Internet and as ISPs, seconds are not that bad but in a data centre in a corporate industry, you don't even want to have seconds in times of convergence. You want to have really, preferably sub seconds or sub ‑‑ a 10th of seconds convergence. And the way to do that is to keep your routing tables small.
The other one is ACLs, and firewall regions. Keep those simple as well. Especially for my friends from the security department, I am working very closely with them because they are sitting in the same room, something like 10 metres apart. Having them have simple ACLs, simple firewall rules, so that they can really allocate simple blocks to specific type of security zones.
Now, specifically for that last part, we split the whole number plan into two main groups. We said if you look generally, you have on the one hand in the network IP space for end devices, and an end devices can be servers, it can be work stations, it can be cameras, bring your own devices, all that. And on the other hand, you need IP space for the network itself. And I'm not talking about the default host, all that. I'm talking about the interconnects, the transit VLANs, the loop back IP addresses. You need IP space for that. And we separated those two into two blocks. And the reason for that is that if you have, for instance, a certain area and in that area you'd need hosts and you have network ‑‑ you need network connectivity. If you take them both from the same block, then that block has got two different types of IP space. And in terms of summarisation, the first can aspect. That is good. But in terms of the third rule, keeping the ACL rule spec's a disaster, because in your whole block you have two groups, because you can't white list the whole thing because you have to white list and then blacklist that little bit of network infrastructure, and then fibre rules with combination they don't work.
So, we said every zone will have both network infrastructure IP space and server IP space.
Now, this is the way we did it. This more or less shows our IPv6 number plan. First, the Ps officially will allocated by Rabobank. Then the entity, IT and the international part and then one for the national part. Then for the allocation, this is the allocation for normal end stations, Z and v, main security zone and security sub zone. Now main security zone ‑‑ I'm going to skip to the next one ‑‑ this is the main security ‑‑ I'll skip back for that ‑‑ main securities, these are the main security zones, numbered from unsecure to high secure and management, always high secure. And each main security zone can have multiple sub zones. For instance we have multiple DMZes because we have multiple types of communicating to the external, for instance, we have a DMZ for customers coming networks we have a DMZ for third‑parties coming in, we have a DMZ for staff coming in and going out on the Internet. And each DMZ is then a sub zone.
So that's the Z and the v. So we can have 16 main security zones and per main security zones, 16 sub zones. That's a lot.
And then the last parts is the label and the sub‑net seriously number. Normally a VLAN number.
So, that a for the servers and if you see it the network infrastructure. You see almost the same way we set it up. But then we have got, we shifted it all down one bit and then have the network infrastructure as the 0 there. So the whole structure used networks and the whole structure used for servers is very similar. It's only the X ‑‑ I'll skip back one so you can compare one ‑‑ here we have three Xs and there we only have two Xs. Well that makes sense because you have smaller groups as well per network, because you don't need that much network infrastructure on a sub zone on a location.
These are the main security zones. And then we can have the security, various security sub depending on how much type of sub zone it specifically is. For instance, DMZ doesn't have multiple locations, it's just a data centre location.
Another thing we did with the Ls and the Xs per security zone, we can divide them over locations. Some sub zones we need on all locations, for instance office or server access. Other sub zones we need on a few locations, and one on all those unmanned locations as well. So that's how we did that part.
Any more questions?
Then Jen I am going to hand over my microphone to you again. Thank you.
(Applause)
JEN LINKOVA: Thank you very much. Now, we are going to talk about very interesting topic related to this presentation and about what's going on with IPv6 in enterprise space because I believe we need to pay some attention to enterprises, ISPs are moving into v6, mobile operators are moving into v6, we have not heard from enterprises as much as I would like to. So now we are going to have a panel discussion.
CHAIR: Okay. I hope my voice is going to last for the whole panel discussion. We'll try to find out what the problems are and stuff like that. I'd like if everyone introduces themselves as short as possible. We'll start here with Enno.
ENNO REY: I'm Enno Rey. The main thing for this panel is I'm involved in a number of projects, usuallily very large enterprises. When it comes to IPv6. And I have different roles both technical and planning, I actually work on the console doing routing stuff. And these are enterprises, say, sort of traditional enterprises, manufacturing, chemicals, ought motive, these type of enterprises so I can probably reflect their specific perspective when it comes to IPv6.
MARCUS KEANE: My name is Marcus keen, I work for Microsoft IT on the network architecture team. We have quite a deployment of IPv6 already, we're trying to move ahead with further deployment, and also removal of IPv4 where we can.
SANDER STEFFANN: I am Sander Steffann. Many of you already know me here from the RIPE community. I am an IPv6 consultant, and do consulting also for enterprises. Like I have done some consulting for example for Rabobank since 2010. So I'm kind of on the border between the ISP world and the enterprise world.
BENEDIKT STOCKEBRAND: Benedikt stock brand. I am a one‑man company. I have been doing that for 14 years or so IPv6. Training, consulting, and I have done a lot of stuff with enterprise customers, not so much ‑‑ well some ISPs as well. Currently in a project where we are deploying IPv6 for an enterprise organisation sort of thing and I'm basically the one who doesn't understand anything in depth but understands how things work together when it Cox to v6.
FRISO FEENSTRA: Fridays owe Feenstra Rabobank enterprise.
CHAIR: I'll start with some very simple questions and see if we have time for the audience to ask questions. What are actually the biggest problems in an enterprise environment? Who wants to start?
FRISO FEENSTRA: The biggest problem in an enterprise to get the focus that people understand that they have to suddenly start something and it should not be considered as a network issue. IPv6, I believe, is not a network issue. In that sense, networks should not have issues. It is other systems needing the network which will need IPv6 at a certain moment. The network should just be transporting and providing communication for end systems. And so, for IPv6, there should be a corporate need within the company for IPv6 and then that corporate need is probably derived from ‑‑ it isn't migration which you have to do one day. Are you sure enough that you still have enough time?
BENEDIKT STOCKEBRAND: In my opinion the biggest problem is not the network, yes, I agree with that. So what is the biggest problem? The applications of course. I mean especially if you are like in a bank or something, you have decades and decades worth of applications and you have to sort these through, except that's still not the problem. The biggest problem is always people. And it's really surprising for a long time I thought the I go biggest problem when it comes to people it management because you have to make them realise that there is some business thing involved that they should actually get going. But I finally learned that a lot of technical people who are just as bad resisting change as their management is. And that's where things get really troublesome a lot of times.
MARCUS KEANE: I can echo the sentiments around the thing with people. I think certainly it's important for an enterprise to get senior level management approval and support for deploying IPv6. Inevitably, with large scale network deployments, there are going to be issues, there is going to be an outage, there is going to be some clients suffering to a degree. So, if there is senior level management support, I think it makes a lot easier for the technical people to sort of, to have that air cover as it were.
SANDER STEFFANN: I first want to echo what's already been said three times, so I'll move on to a different one. I have also done a lot of training, for example, at Rabobank, and what's always the most difficult bit is all the different options that v6 provides. V6 is a great protocol. You can do a lot of things with it. But then you are doing training and you actually have to explain to people what all the different options are, which direction it is moving, and how they all work together and at that point you see steam starting coming from people's ears, so, you know, just a simple thing that we have all faced, SLAAC versus DHCP. A bank goes no we just want to do DHCP. I say that's a fair choice so stop using Android phones because they don't do it. In that case we'll do everything in SLAAC or RDS S. That's great but count on your wind owes machine not getting a DNS server. We dodo this, and that, combined with that to actually get a situation that covers everything. And just the complexity of the whole process is the biggest technical challenge I think. I agree that the management challenge and the people challenges and the people not realising it's not a network problem, those are the bigers ones. When you get to the technical level I think this is the first one that POPs to mind.
MARCUS KEANE: I can just add to that actually, there is some technical pain as well. The reference earlier on to keeping the access list simple. But the biggest problem I think you could face and we face is trying to keep v4 and v6 policy congruent. That's a major headache where you sort of particularly if you get network changes that suddenly what was working is now not working, and the help desk can't seem to figure out why. It's critically important to keep on the routing level and the security level to keep v4 and v6 congruent.
ENNO REY: I can echo most of the stuff the other guys said but I am playing a bit of devil's advocate here. What we see in these type of industries that I mentioned during the introduction, is we don't even get to the point where like SLAAC was DHCP is discussed. For many of them, there is still, at least for a large part of the network, still no compelling use case. It is ‑‑ I mean we can all lament on this, but it is like that, it's ‑‑ most people we interface with have realised it could make sense to enable public facing systems with IPv6 and pretty much everybody that I know, and the environments where we work, does this in a dual stack way. We don't see in that type of environment any v6 only. But, there is some insight, okay, this should be done. There is some industry which was realised okay, it could make sense, we usually ask questions, do you have an IOT product? Then it could make sense from the product side to think about IPv6. Do you have an AP for something like for an insurance company? Then it could make sense to have IPv6. But once those two are discussed, I found your presentation frizz owe yesterday very interesting on the angle v6 for identification and for compliance in a certain sense. As I wasn't aware of that one. With you once you run out is there an IOT or is there an AP, it becomes very sparse when it's about convincing senior management why should we do this. And this is, we still see this a lot. So, combined this with bringing senior management support in or it's not just a network topic, it becomes very difficult to convince people still on why v6 should be there in say their client or their production networks.
BENEDIKT STOCKEBRAND: When it comes to convincing management or people, basically you have to take a look at the business side. I have had one customer and they were really really excited about IPv6, not because of IPv6, but because of Microsoft. Because of direct access more specifically. I mean you can kind of use direct access in combination with IPv4. But it's just unnecessary pain and they actually realised that, smart people, so they wanted to have IPv6 up and running for a lot of things so they can have their field agent sort of people use their access for whatever, and with whatever network they found on the spot. That's a really important thing. And the other aspect of course is, another customer of mine, they realised we have increasing problems with people being stuck behind DS‑Lite when it comes to VPNs and similar things. They decided okay, it actually obvious, even at management level, that we need to do something about this. And the only way to deal with this was to provide the external interfaces with IPv6 as well as IPv4. And that customer basically we came up with a risk driven plan and surprisingly management and technical people actually, they used different words but they think the same thing. And we came up with this thing, first thing we have to take care of that people who are using outside networks that are not under our control can access our services. Then we see that they get IPv6 ‑‑ or that he can deploy IPv6 in the internal networks where we need them, for example for things like direct access on the internal server side. And then we see about getting rid of IPv4. Because, this one thing, and I think Enno and myself, we seriously disagree on this, but just doing all and everything, dual stack, just adds lots of complexity and constraints that make life miserable for years and years and years. And trying to avoid that is one of the things that's difficult but it's really well worth it.
ENNO REY: I'd like to very quickly comment on this. First of all, I did not necessarily recommend it. It was an observation. But, the second thing is, I agree with what you said on remote access. But, unfortunately once ‑‑ and we have seen this in environments too, the dual stack light problems and then to have proper VPN working and enable IPv6 on the VPN gateways ‑‑ once that is done, everybody is like, oh, okay, now it works, so we can get back to real projects, the stuff that we need to in the next 6 or 12 months. And isn't it ‑‑ one could ask, isn't it sad that the actual driver for IPv6 in such environments is ‑‑ well, we have to fix something that is broken without IPv6 so as not to real like we gain something but we fix something because providers gain something from IPv6.
BENEDIKT STOCKEBRAND: That's one of the sad realities of life. When do people go to the dentist? When they can't stand the pain any more. And yes, you are sitting there you know they will be in pain like a couple of months a couple of years from now, but still, these very ‑‑ you have very few chance to say actually convince people to take care of things before the pain hits. It's frustrating, but that's just the way people are.
ENNO REY: But once the pain is fixed do they live more healthy from then on? And they don't.
BENEDIKT STOCKEBRAND: You are an optimist there.
MARCUS KEANE: I can strongly echo that in terms of, I guess for larger networks with growth, you know on the one hand IPv6 can be seen as panacea, but if you still have to deploy dual stack, it really isn't. Is really it will become very interesting when it becomes IPv6 only, because dual stack has been said many times before, is really the complicated factor with deploying IPv6. It's just the interactions in terms of support, on the help desk and so forth. So I think that to me would be the complexity. But if we can get over that hump, I think things will be a lot easier to manage. And then we can sort of concentrate on just running networks rather than battling addressing schemes and so forth.
FRISO FEENSTRA: Currently for our internal networks, for this year, it's planned to start the migration for our internal network towards IPv6 and I think that project will last something like three to five years, and that is the project of bringing the complete internal network to dual stack. Only once that is well underway and we have not only the network on dual stack but also the platforms on dual stack so that number of applications will start using IPv6, and once a number of applications are well running on IPv6 and we can see that specific platforms can do everything IPv6, IPv6 only, only by that time we will start switching off IPv4 on specific networks where it's not needed any more. So, I'm afraid that we are in for a lot of time going dual stack and to be able to manage that time, we will want to keep our IPv6 implementation in terms of routing, firewalling, topologies, very similar to what we currently have in v4 just to keep it manageable so we can tell the people at the help desk at the first line, don't worry about the IPv6, it's very similar to what we currently have with IPv4. And I think that it's only that way that we can sell the whole project, specifically within our own department, because as said, even within our own department, there are enough people saying well should we do this? Why do we have to complicate it? We are busy enough keeping the network up and running at this moment. Why make this additional complication?
ENNO REY: May I ask a he' quick question here. You mention it will be like a three to five year effort, and you use the term project. So there is a project with a sponsor and a budget for like IPv6 deployment or is it split into several parts?
FRISO FEENSTRA: It's going to be split into lots of parts because the network ‑‑ I think that if you look at the Rabobank IT organisation, you have two main parts. One is infrastructure, and the other one is called IT systems, more looking at the applications. And in terms of infrastructure, we will need support for almost all infrastructure support groups, whether that be the various platforms we have varying from Windows, Linux to Z‑OS and nonstop systems, so middleware, to printing, to office, to Cloud implementations. The whole programme of going dual stack will involve the, at least the whole of IT infrastructure with ramifications into IT systems, the applications. So, indeed it will be a big programme, it has to be sponsored on senior management level, specifically if it lasts that long.
CHAIR: All right. We got carried away already from the question. But that's fine.
So would any things that you happened you didn't expect with integrating some v6 in corporations, some unexpected things, what were they if there were?
FRISO FEENSTRA: Everybody is looking at me again.
As I said, we did it on the external, and what I said, we wanted to deploy IPv6 in a similar way with IPv4. Now, one of the nasty things we came across is that on specific elements, there are indeed differences in the implementation of IPv4 and IPv6. For instance, we had a BGP session, a multi‑hop BGP session between one part of the network and another part of the network, and that multihop BGP session was generated from the network interfaces towards the intermediate stuff, but those network interfaces, they were using H S R P, Cisco implementation similar to VRRP ‑‑ and with IPv4, if you have BGP, then it is the IP address which is used as source, the source IP address for the BGP session is also the physical address. But with IPv6, the source address was the virtual address which would move over from one device to the other device. So, then suddenly if you would failover, then the whole H S R P wasn't working any more, so we had to revert the multihop eBGP session not from interface but from loop back interface. And that was a bit of a, wait a second, what is going on? This is not as we expected. Let's revert this change and go back to the drawing table and see exactly what is going on. So, this type of, wait a second, implementations can indeed be different for IPv6 and IPv4.
ENNO REY: May I ask, when was that? Like, six months ago? Three months ago?
FRISO FEENSTRA: A year ago.
ENNO REY: I would like to jump in here, that you were wondering like, you were probably wondering, are we the first ones who toen counter this? And we have this type of situations on like BGP route maps stuff, features and firewalls behaving a bit differently on something all the time in several environments, and the frustrating thing is, there is many of those situations, wait a second, somebody else should ‑‑ must have encountered this before and you approach a window and it's well, no, you are the first once to bring this up on standard equipment (ones) Cisco, whatever type of major firewall Windows ‑‑ and this is frustrating. And it emphasises, unfortunately, that you have to test everything. Never ‑‑ you shouldn't assume if I may provide some very trivial advice here, you have to test. Don't believe any Windows statements. Don't expect ‑‑ like it has worked with v4, so it's the same configuration, just with a v6 address. And even, we had cases where like, a command was accepted from a device and didn't do anything for IPv6 and then you approach a window and open a ticket and it's frustrating to see this but it emphasises testing is level needed.
MARCUS KEANE: You can enable stuff and it's even worse if it does accept the command and it does work, but it's being processed in software.
BENEDIKT STOCKEBRAND: My biggest surprise was actually completely different dimension. Basically, I was in a project with IPv6 and for some reason we actually managed to convince people that this is not a network project, so we had software developers and procurement and all sorts of people involved there, very nice, because afterall, IPv6 in an enterprise is not a network topic and we got started so the first thing you do is set up a test environment and the first thing you need in there is obviously networks, so you have got the network people, try to explain to them what they are supposed to do and what they are supposed to run and then you bring in the software and try the software, only the software developers in the project are gone, because after all, it obviously is a network project. And I never would have thought that this would take us so much of a surprise, but seriously, we only realised the software developers were gone to, assigned to different projects by the time we had the network and the infrastructure ready for them to test the software. So, one thing I have learned there, make sure you get the timing right when to bring which people into the project.
CHAIR: Okay. So has there been anything you haven't been able to solve?
BENEDIKT STOCKEBRAND: People.
ENNO REY: I can give a technical answer to this. Some of you might recall we, I gave a talk in the v6 Working Group in RIPE 72 in Copenhagen a year ago, on, from an environment where we wanted to implement LL A base, so link local addressing only for the routing protocols. And that case is still open with the window, it hasn't been solved since a year and even though it's a large customer and the window in question, which is I mean if you look in the archives at Cisco, there hasn't been too much movement. So, yes, there is things in that environment we would like to implement and we still can't do this.
SANDER STEFFANN: Basically, what you are saying is, it could work but we just can't get the vendors to deliver or?
ENNO REY: Yes, exactly. Even from an environment which has several 10 K of Cisco devices.
BENEDIKT STOCKEBRAND: The most annoying, frustrating, hopeless problem I have run into so far is business specific software. So basically, the crazy story about this is that I had done a training on IPv6 and one of the participants eventually shot up, yeah, I just called our software vendor for some blah, blah, blah whatever specific software, and asked about IPv6. And also, you don't need that, yes, because we want to blah blah, no, believe me, you don't need that, was the response. And that can be quite surprising, because a lot of people think okay, what do we need? Well, the product is Microsoft. Okay, about you that stuff works, I have been told. It's really the special software for specific purposes that are a real pain and you have very little control and very little leverage to actually convince the manufacturers to deliver IPv6 support. And that can be a real show‑stopper that you want to be aware of as early on as possible. Because sometimes your only option might be to have the specific networks where this is used dual stacked or use some sort of virtualisation so you can keep IPv4 out of the end device networks and run IPv4 on dedicated servers so people can still use that software. You really want to be aware of that early on, and ‑‑ well short of having ‑‑ there are very few ways of solving that.
FRISO FEENSTRA: I think that we anticipate that as well in our project, what we have, what we were planning that, that's one of the reasons why I'm saying we will be dual stack for sometime. If I look at own infrastructure, for instance quite recently we have hooked up things with security cameras, burglar alarms and fire systems and even bathroom toilets to our network, and I'm happy to say yes they can communicate now on IPv4. I don't think you will get them to work on IPv6 for the coming years. It has taken them something like 25 years to come from the old scanner protocols into the world of IP. They just arrived. They still got their own IP protocols where you think why don't you use standard TCP IP, no, they have got their own IP protocols. I'm not yet there. I don't think that they will be very quick in adopting IPv6.
SANDER STEFFANN: You are steel asking about things we haven't been able to solve or surprises. Just reminded me of a story that Ron abusers ma, from the US defence told me. He was setting up a lab and a vendor came in and was like we're going to set up the lab, what's the management IP address. He was like 2001 colon. They said hold on, that's not an IP address. He sends them back, they come again, we fixed it, we can now manage it over IPv6. So they set up the lab everything works and you think the problem is solved. Until they wanted to re‑number the network and he was looking through the web interface and through the CLI he was like where do I change the management IP address. Call up the vendor, he goes like we have to send the software developer over, we hard coded that.
ENNO REY: On the bright side of things, for a positive note, I'd like to mention that when it comes to like enterprise software as IP stuff for example, IPv6 support has gotten much much better in the recent years. There is all these talk like well look at the applications and so much stuff is not working over IPv6. I don't share that point of view any longer. I would say, when it comes to enterprise software, things have gotten much better, especially like SI P stuff or so, like in the last 24 months. So there is real movement in that space. You don't have to focus on that aspect so much any longer.
BENEDIKT STOCKEBRAND: I think actually we are getting close to the point that we can actually tell management okay, we have these and these and these are residual problems. There are still IPv4 only and we can put a price tag on them because that's really really helpful when you have to talk to management, aside from wearing a shirt of course. If you can put a price tag on things because there are just a few, fine. If you basically know, okay, we have ‑‑ okay, Microsoft Office suite does IPv6, but everything else doesn't, it's very hard to tell management it actually makes sense to go this way. And we are making rapid progress in the enterprise world, with this, fortunately, but it's taken long enough.
CHAIR: What are the biggest differences between enterprises and ISPs with integrating v6.
BENEDIKT STOCKEBRAND: People.
ENNO REY: Try to give a quick answer. There is two aspects. One is probably its complexity or heterogenity, when it comes to the environment. We have one customer, which actually, this is a serious number, they have 100 K plus applications in their whole space, grown over years. I know that, I have been working in carry environments I know they are complex, but that not complex. The thing is its much easier to say sell IPv6 once the main product is IT related as people have a different mindset there. Oh we need this to deliver our product, but once the product is like chemicals or machines, it becomes much harder to convince like, you need IPv6.
MARCUS KEANE: I can second that. I mean from the scale, our network is probably the size of some ISPs, but the problem is exactly that, if you are trying to support the business in various business groups, that you end up having these bespoke networks which make is quite complicated to manage. So I think that's probably one of the biggest differences between. If we were a regular ISP we could mandate a standard looking service, but unfortunately, the business owners require sort of different things at different points.
SANDER STEFFANN: Yeah. Mean I am simplifying it a little bit. If you look at an ISP, providing playing pipes. You transfer packets. You have problems with the CPEs and management platforms. There is a lot of stuff that an ISP has to do. But in essence, you are providing open pipes that transport packets. If you look at an enterprise, you are dealing with applications, firewalls, different departments, different requirements, the whole complexity, the whole set of issues is completely different.
So, I think it's really hard actually to compare them.
CHAIR: Okay.
FRISO FEENSTRA: I think that for an ISP, IPv6 is much closer to their core business as with an enterprise. For an enterprise, the enterprise and the field they're in, that's their core business, and to support the core business, there is IT and somewhere at the remote end of the IT, there is network, with the migration from IPv4 to IPv6 is. It's very much in the remote. It's not core business. With an ISP, IPv6 is much closer to core business.
BENEDIKT STOCKEBRAND: And when I say the problem is people. I'm serious about that. In a number of ways. And what I have seen is with the ISPs, the ISPs I have seen so far, very few exceptions most which probably don't exist any longer, or working hard to make themselves nonexistent. In an ISP context you normally have some people who know what they are doing. And in an enterprise context you can actuallien counter network teams, doeses of people, none of which really understand encounter) how TCP works or whatever else. They have the cook book recipes all right and they get things done that way, but if you introduce something new that is somewhat different where they don't have experience with because all they do is work from experience not from understanding, then IPv6 can get really, really scary for them. It's a completely different mindset, completely different level of people you are working with. And down to the point that you have to explain the network team that it's a bad idea to configure the MTU F interfaces consistently throughout single VLAN. And I'm not making this up.
CHAIR: Okay. Thank you for all these interesting comments. Yes, there is audience who have some questions.
AUDIENCE SPEAKER: Wilhelm Boeddinghaus, iubarii, also doing some IPv6 consulting, I have a comment on ‑‑ maybe other incentive why enterprises should go to IPv6 and this is the vendors. They took a long time to pick up any interest in IPv6 and they sold licences for IPv6. And as soon as IPv6 reaches 50% in the Internet traffic they will lose interest in IPv6, you will ‑‑ in IPv4, you will get the attaches later, you will get features later and in a few years time you will pay licences for still using IPv4. So this is a big inincentive to go. This is just a comment. And I have a question for the panel you all say dual stack is cumbersome. So why can't we have IPv6 only in the client networks and run IPv4 as a service over it? We do it in mobile phones in the US, 464 XLat it's called, we have maybe other technologies, why can't we do this in an enterprise environment and get rid of dual stack?
MARCUS KEANE: I can answer that. So, 464 XLat works fine in Android, but that's a single platform. In an enterprise network we have many platforms. So for example Windows does not have 464 XLat or anything like it. There is also a philosophical argument about that as to whether you were hiding bad software practices. So effectively what you are doing is you are covering up for people hard coding v4 literals into software. So, yes, you are making life smoother for them but RIR furthering the progress of the network in any way? And it's arguable that you are not. So, for an an enterprise point of view, there is a lot of things that could break going into IPv6. So we have been discussing the idea of moving, to v6 only in the guest network. We have now sort of done an about turn on that because we realise there is very little generally VPN support for IPv6. So that remains, or least us with the corporate ‑‑ the internal corporate network. But again, there has to be a lot of telemetry to figure out what is actually being used in the network, so that if you do withdraw on v4, that you are not going to break a lot of people and make them unhappy. So it's a slow progress, if that answers the question.
CHAIR: Thank you. We have one minute per person from now on. So...
AUDIENCE SPEAKER: I am from the RIPE NCC I have a question on Jabber from Marco, so for the Rabobank, will they put pressure on Dutch ISPs to enable IPv6 for their users.
FRISO FEENSTRA: No.
CHAIR: Thank you.
FRISO FEENSTRA: We are in contact with them, we are working with them and they have got their own drivers for moving to IPv6. We're not in the position that we can very strongly influence them. They have their own drivers for moving to IPv6. And I am trusting in those.
AUDIENCE SPEAKER: Natalie Trenaman, IPv6 programme manager RIPE NCC. I asked this question in every training course for anybody who did an IPv6 deployment, because management wants to know. What was the most expensive part of your IPv6 deployment? Where did you have to spend money.
FRISO FEENSTRA: Training.
BENEDIKT STOCKEBRAND: Training.
SANDER STEFFANN: I was the one being hired to do the training.
STENOGRAPHER: You must be loaded so!
MARCUS KEANE: Most expensive. I think a lot of our deployment, we have been sort of deploying IPv6 since 2006, so, I think a lot of it has been incremental as you sort of age out devices and so forth there is obviously been a concerted plan to build in v6 when you are replacing devices. Perhaps in terms of complexity, again it comes back to sort of ensuring that congruence between v4 and IPv6. That definitely takes longer than if you have just one protocol.
FRISO FEENSTRA: I want to explain a little bit about what I said in training. One of the reasons between the success of our project within Rabobank, is that we got quite a large community of our IT staff on the project by providing them the training and providing them information on what is exactly going to happen, how are we going to do it? What does it mean? How is it exactly set up? And in that cents, we had I think something like 15 sessions, and each session was attended by something like 20 participants, so we had over 300 participants in our IT organisation getting a lot of information on what this IPv6 would mean for them and how it was set up. And I think it was, part of the success of the programme that this information was indeed spread over the whole organisation, and I think yes it was one of the most expensive, not so much in terms of the person we hired to do it but more specifically in time used for all the various participants, because participating in the training means non productive hours on their regular jobs.
SANDER STEFFANN: Really, one of the things I noticed while giving the training was that because I had participants from software developers to security to every department, I noticed that a lot of them not only learned about IPv6 but also, in general, about networking. So I think the benefit was even beyond just v6.
BENEDIKT STOCKEBRAND: And one more thing about this. You want to do trainingings early on for people throughout the entire organisation. You need spies ‑‑ multipliers. Because those are the people who know about the environment and they look out for things that might be or might not be v6 related. They know you personally, they say I think I found something we should think about. And you want that as early on as possible.
AUDIENCE SPEAKER: Jan. Bloomberg. A question for consultants. So in all of your consulting advent users, have working and potentially recommending ways of doing greenfield builds, have you discovered whole new ways of doing network designs? And we're talking general network designs, because IPv6 changed the way you design networks.
BENEDIKT STOCKEBRAND: Yes, you do. Much smaller sub‑nets, much simpler security because you separate things that are allowed to do different things into different sub‑nets, and that's why dual‑stacking can be such a problem, because if you dual stack you have to keep in mind that you want that afterwards and you don't want to build up a legacy of IPv4 driven network topologies, but yes, IPv6 can make one hell of a difference especially if you have large end user networks. The data centre it frequently doesn't make that much of a difference.
ENNO REY: A quick answer from my side. LL A only addressing for infrastructure is an I think would be a benefit for mainly networks I think. Other than that. No. Nothing identified.
AUDIENCE SPEAKER: Hello, Arab owe fair bank. What I heard here is mostly about big enterprises, what I could see for my for having clients as very small enterprises is another factor which is, was not clearly named, it's the external contractor which often comes and tells us they are off IPv6. I don't know what is it or I know what is it or I don't know how to handle T please turn it off, I don't want to deal with it. This is something that has to be fixed and personally I don't know how.
MARCUS KEANE: NAT 64.
BENEDIKT STOCKEBRAND: Give him DS‑Lite and let him feel the pain.
AUDIENCE SPEAKER: Blake Willis from Zayo and I'll be quick. Zayo: This is an organisation had a that is done 40 MNAs in 40 quarters and I recently helped our IT people develop their IPv6 rollout plan for internal telemetry and corporate networks and the main driver for it was alleviating pain of IPv4 address conflicts. Some quick take a ways. One was try to keep your sub‑nets on bit boundaries. And we ‑‑ I intentionally kept kept everything basically was on a very clear bit boundary and made the correlation that 40896 VLANs was about the biggest number an IT person could handle and pretty much kept it to that and human readable numbers and managed to reach consensus on this. So just to share an anecdote.
CHAIR: Right. We're done. Thank you all for being on the panel.
(Applause)
JEN LINKOVA: Before you all run away to deploy IPv6, I have three things. First of all please rate the talks. Second thing, two announcements from PC. A PC election, voting is open until 5:30 p.m. tomorrow. So please vote for candidates. And the last announcement, as you might have noticed, we don't have RIR report till Friday morning this time. Instead, 9:30 a.m., yes, Friday 9:30 a.m., I know it's early, there will be a session about how can we increase diversity in RIPE meeting community, so please do your best to wake up on Friday morning and attend this session.
See you tomorrow.
(Coffee break)
LIVE CAPTIONING BY
MARY McKEON, RMR, CRR, CBC
DUBLIN, IRELAND.
