Database Working Group
11 May 2017
2 p.m.
CHAIR: Greetings. Welcome to the Database Working Group. We have had a lot of exciting things happening since our last meeting. Obviously we have changed all of our chairs over. I am William sill vest err, we have David holiday Aero and Denis unfortunately was unable to be here today. At this time, we'd like to designate a scribe. I think Alex from RIPE NCC has been so kind to offer. So thank you. And Nigel you are off the hook.
So this time we'd like to accept all of our notes from the last meeting. Does anybody have any comments or any objections to accepting the minutes from the last meeting. Seeing none, we'll consider those accepted. We have got a full agenda today.
And with that, Tim I'll let you come up.
TIM BRUIJNZEELS: So. Two presentations for me today. To start with this one I'll try to keep it short. This is the database operational update.
So, releases of the core WHOIS database since RIPE 73 have not been many. We released number 1.88 shortly after the last RIPE me, it was in the release candidate environment at the time of the last meeting. It introduces no major changes to the data model or the way you interact with the database. But it fixes some issues that, some smaller issues like you are able to delete objects for example that lack a mandatory attribute. Beforehand you would to fix the object first and then delete it. And actually also in our way if we needed to clean up objects. And there is the status value of resource objects that is fixed. However, if the value is not set, you can now change it to something proper.
Then I'll get back to this one on the next slide.
We recently had a bug fix release to address a memory issue. And finally, we had a new version in the release candidate environment now where we did two ‑‑ the most visible things are, we now include information on who updates an object in notifications. So, and I realise actually that's quite hard to test in the release candidate environment because you won't receive e‑mails from there. But I can send a template to the list after this session.
One thing to note there cannot do this for maintainers. We can only tell you which SSO account or which BGP key was used because if we did for maintainers, that could potentially leak information about the same password being used on different maintainers on the same object. Because we only have the hashes and then we wouldn't know which maintainer was intended to be used. So, that's why we don't tell you this maintainer was used because it matches that password that was submitted.
I hope that's clear. If not, then there is questions at the end and I'll try to explain better.
And another thing. We collaborated with our DNS folk and we replied the DNS check software for Zonemaster, so when you submit domain objects, the state of the name servers will be checked using new software. And that's also alive in the release candidate environment now, so if somebody wants to have a look at that, that would be grand.
We think it's better, our DNS people think it's better, so it gives more clear messages.
Now onto the not so nice slide.
People who watch the mailing list would have seen this before. We had an incident on 26th November last year. This incident was later found in a route cause analysis we found it was due to memory, over usage, caused by an issue in the shared cache. So between our different nodes we keep a shared cache of access from different IP addresses in order to do rate limiting on personal data. Now, we didn't have an excessive load of queries that day, but we had a load from many different source addresses and that caused an issue there.
The quick fix then was to disable the shared cache. We increased the memory. We re‑enabled the cache, apart from extending the memory, we also extended our monitoring of it even further, and when we saw one node having some issues, starting to have some issues, we were actually able to look deeper inside the code where the problem was and were able to put a fix out for that.
Unfortunately, in our efforts to get the memory extended, we had an issue when we deployed new servers because the mail was not configured properly on them. So when it became part of the cluster, mail updates, you would have received an e‑mail saying that there was a failure.
But once identified, that was quickly resolved as well. And that's actually all I have on this slide deck, but of course I am willing to ‑‑ I am open to questions, comments or anything you want to discuss about the operational side of things.
Okay, so I'll go on with the next one.
Then on my previous slide like I said we only had a few releases of the core WHOIS but that doesn't mean we weren't doing anything, we have done a lot of work on the usable side of things so why is this an issue? We have talked about this many times before but just quickly.
We feel that the RIPE database is very complicated and in fact too complicated for many incidental users and also if you look at the interface that is we provide, they kind of provide all this complexity to the user. So, they can be quite daunting to use. And a while ago we already started to make things better, here is again the reasons why we want to make it easier to update things.
If things are very hard to maintain, then users give up. Sometimes they send us a ticket, but in many cases they would just not bother. And this has of course an adverse effect on the data accuracy of the database, and its usefulness, so that's an a big deal.
So, a while ago we already started with things like we introduced a default maintainer for LIRs so you could maintain your organisation, object and your INET NUMS in the database rather than through the LIR portal with object editors, 72% of LIRs have enabled this. You might only need to enable this when you want to make an actual change, so since our introduction a lot of LIRs have actually done this.
We already made a start with the web interface for updates a while ago, so that now has auto complete and it's all centred around using SSO accounts for ease of use. But you can still of course do something like you know create an object in the text area if that's, you are that kind of user.
We took this a bit further when we looked at reverse DNS because that was actually causing the highest load in terms of tickets and people confused or giving up in fact. So, now when you say create domain object, we first give you a warning about all the things that you need to get prepared first. And then we created a wizard like interface to help you, guide you through the process. So we need like a prefix, that prefix needs to be okay and you need to have it. Now this is not in the real database, so, I mean I wouldn't have access to that prefix in real life, but you know, on the slide you can make it work.
You then need to specify name servers, at least two of them, and we do quick checks already whether those name servers are actually there and they are authoritative for the domains.
Another thing not visible here but that we dodo, is that you don't need to figure out all the 24s domain objects that you need to make, for example you want to set up reverse DNS for a /20. We will in fact create all those objects for you in bulk.
And all in all this led to a massive drop in the error messages that we saw, so it's about one third of where we were before, and those are usually real issues with the name server that are not caught by the quick check up do here, because after these checks, it's handed off to the core WHOIS system and then we get that interaction with Zonemaster that I mentioned before, same as if you would send a domain object through another channel like mail updates or sync updates.
So if there is still a problem there, then you will also see it.
Then, recently, this is now in the release candidate environment, we have introduced something we called my resources. So when you are logged in and if you represent an LIR, we can actually show you your, let's say, top level objects, so your allocations and assignments that you got through the RIPE NCC. And what's more, is that you can look at the details of these objects and for example for PI, for an allocation, you can see how much of the space is Telia signed and you can look at the more specific object load. You can click on those to recursively go down as far as the objects exist.
As I said this is in the release candidate environment now, so we would really like you to test. We have already lacked at it with a few people individually but it's really valuable to get your feedback. In time it's important to mention, we are actually looking to remove some of the boundaries between the LIR portal and the database in the sense that now you really need to move around, navigate around quite a bit, which is quite ‑‑ is not very user friendly. So, eventually, we'd like to show you both together essentially, so if you are an LIR you'd see LIR services but you can also see your objects in the database. And that means we'd like to deprecate or phase out certain things that are currently available in the LIR portal but only once we are satisfied that all the functionality is needed is there.
So, also on that front, please have a look at this and let us know if there is anything you are currently using in the IPv4 and IPv6 analyser for example that you don't see here and would like to see.
Deployment is planned together with the release that's of the core WHOIS that's in the release candidate environment, and it's planned for 22 May. So that's not this Monday, but the Monday after.
Longer term. Well, time permitting when we can work on this, we can actually also open this up for non‑LIR users because if we see that you are connected with the maintainer that has other resources, we could show that. So there is no reason why you couldn't use a similar interface. Of course if there are LIR specific or member specific things, I should say, then you wouldn't have access to those, but the idea of being able to look at the II K of all your objects is still useful
Other things you can do based around this concept you can think about easily to facilitate both changes of resource objects for example to manage contact information or to manage authorisations, maybe you are using many different maintainers and objects and you'd like to consolidate this or maybe you want to do the opposite and delegate things.
We can also start to highlight issues that we see. Like the wizard on reverse DNS that I showed earlier only works for create. But we can do stuff like show you the current state and if there is any reverse DNS Lameness we can highlight that. We could do things similar to what we currently do for the, if you want to create ROAs in the RPKI interface we tell you these are the announcements that we see you do, and these are the ROAs, mention them. We can do similar things for route objects and of course we can highlight other issues such as syntax problems and this actually exists, assignments that were made on other assignments. That's not supposed to happen but it has happened in the past so there is some cases like that.
But first, I want us to talk about this one because the query side of things also needs love. The RIPE database query interface exposes all the options that you have at your convenience on the command line. So, that's a lot of options. Quite daunting for a lot of users, and actually we are monitoring this. We see what people use, and this is an actual example. There are people who treat this interface like they would sirrey and they go daddy, tell me what the peering policy is for level 3 and (DB) they don't get a very level answer. It just indicates that I think the type of user that we get here, typically they don't use any flags and when they do use flags it's often something that's enabled by default already, like show full objects details, I believe.
So, there is a disconnect here between the type of user that uses this and you know what's provided. So, we want to make it easier to find things because the easier it is to find things, the more useful this database is out there.
Analysing the usage patterns and we want to support the common cases much better. Like I am interested in finding the contact details for a certain resource for example. I think that ‑‑ we are still shaping how that will look but it's something we'd like to work on in the near future.
And of course it will still be possible to do more advanced queries as well. The input field actually supports all the options that you use on the command line so if you go here for convenience and you know how to use the options. You can always do that (command)
And that actually brings me to the end of this talk. So these are the things that we have done and have in mind for the future and, again, please have a look at the release candidate environment and let us know what you think. And with that, any questions, comments, I am happy to hear.
No questions, okay.
CHAIR: Thanks Tim. So, over the last few months we have had a challenge with your mailing lists and it's popped up that we have had issues with users not receiving e‑mails because of D mark. Raise your hands, does everybody know what D mark is? If you don't know what it is raise your hand. So D mark is basically a new standard in the IETF that's designed to help ‑‑
RANDY BUSH: It is not a standard.
CHAIR: Proposed. I apologise. It is proposed and been adopted by several service providers that are out there, primarily right now AOL and Yahoo. G‑mail has announced they are intending to support it but have not yet. Unfortunately there is a little bit of a conundrum if you support D marks you break other RFCs that already exist. You probably noticed that we have had the mailing list a few months back changing the headers on all of our e‑mail. That's created an envelope that has changed the interface for how people use it. And this is something that the chairs today talked about for all of RIPE because there is a problem because all of RIPE. We sort of feel that it is something that we can't hide from. But at this time I'd like to open up the microphones and see if anybody has any comments on both either solutions that we should consider, do you support the changes that we have made on the mailing list? Do you absolutely dislike them or anything else? Does anybody have any comments? Concerns?
AUDIENCE SPEAKER: Peter Koch DENIC, so so this is ugly. What brought me here was you mentions that the chairs talked about this for all of RIPE. So what is the community going to expect and do ‑‑
CHAIR: It hasn't been determined yet. The Database Working Group has been sort of the test bed for this. Primarily because we identified some of the problems. We had collated that adds challenges within RIPE not even understanding the issues at the time. As it stands right now, RIPE NCC is looking into this. They are trying to identify some solutions. The mail list software we use is called Mailman, we have made some upgrades and the Database Working Group has been a pilot testing out some of those features. So as a Working Group I have soliciting input you know from the members because this really affects you guys.
PETER KOCH: Thank you. The IETF has been suffering from the same thing I guess, which is interesting given that Randy pointed out that this isn't an IETF standard. To some extent it seems to be a reality. I don't like that and I would like to see a solution that limits the damage for those people who don't contribute.
RANDY BUSH: I'll put it a little more strongly. It is not an IETF standard. It is highly contentious in the IETF. The IETF mailing lists have this problem seriously. There is a significant part of this population that says when it causes trouble, throw the crap on the floor.
CHAIR: Great. Any other comments or feedback? All right, well we'll leave this open on the mailing list ‑‑ Peter?
PETER KOCH: Sometimes we work on numbers. Like, operational data and so on and so forth. So somebody obviously had a problem there like not being able to receive the mails from the mailing list, like Dr. Whenever I use D mark it hurts so don't do that. D mark with a K ‑‑ that's great, that's memories ‑‑ wow ‑‑ advice to the speaker, never read the transcript. So how many people roughly were affected and how hard would it be for them to get a real e‑mail address.
CHAIR: The stats as I was told today, there's roughly 550 people across all RIPE mailing lists that fit into the AOL or Yahoo bucket. There is substantially for in g‑mail which isn't a problem yet. So the question really comes down to bad user experience for everybody or excluding some those users, and part of the challenges, any time someone sends an e‑mail from one of these D marked e‑mail addresses, it doesn't make it to the mailing list. Now, often times it will make it into the archive although we notice sometimes it didn't. So the question is, is the mailing list working for everybody? Are people finding that they are having issues with the mailing list? We have noticed that participation is low. We are wondering is this part of the contributing problem or if there is other issues which is why we put this on the agenda today. I'd encourage anyone having issues or concerns to either e‑mail the chairs or feel free to bring this up on the list as RIPE NCC looks to move forward we'll of course provide updates which I'm sure will be presented both on the bigger list from RIPE, but then we'll also provide updates within the Database Working Group.
Seeing nobody else at the mikes I'm going to turn the microphone over to David who is going to go over our work items.
CHAIR: Okay. The NWI is the work items for the Database Working Group. So, there was an e‑mail sent to the list actually a long time ago with the list of NWI, and came out of messages feedback participants, and not much has happened since then. So I'll quickly go through them. And the idea is that either we generate discussion here now and later in the list, or we decide if there is no interest in any of them, we just drop them. We need to check with you guys.
So the first one is staying on top of abuse contact changes. This meant that if you have an Abuse‑C and you have a role object, somebody changes it, if you have not set up maintainers correctly here and there you won't receive a notification that your abuse e‑mail address has changed you are unaware of that unless you query the database.
Displaying the history of objects, that's something that was there since the history flags were actually introduced. Due want history of person objection, is it possible to do that there has not been much discussion on it.
The AFRINIC IRR. AFRINIC set up their own routing registry. There was discussion about kicking out the AFRINIC object from the RIPE database to AFRINIC once and for all. This opens a whole lot of different kinds of problems as it will essentially make the RIPE Internet registry a RIPE centric Internet registry kicking out all other resources.
Number work item 4. This one is an interesting one as well. The RIPE database has a limitation that INET NUMs can only be parent /child relationship with a key, you are not make an assignment as the same size of your allocation. So if you have a /22 and you want to assign a /22 you will need to make two objects, cannot make only one.
And then out of region Route‑6 aut‑num, that falls with the IRR. What to do with them, if we decide that the RIPE database is not for Internet ‑‑ II R from other regions, they will be left overs of Route‑6 and aut‑nums in the RIPE database.
6 is that every time there are changes made to the RIPE database, how to convey those changes to the world. If a mandatory attribute is tomorrow made optional. How many systems will breakdown because of that because it was declared mandatory, now we make it optional, we delete it, whose system will breakdown and if we make those changes how to make people update their software.
And number work item 7, the Abuse‑C implementation which was mentioned this morning in the anti‑abuse Working Group. Ever since this introduction it has been a little bit problematic. People have different views on how it should be used and how it should be maybe changed to make it easier for people to use. We have not received much feedback on this one either. It's the only one that is slightly active at the moment. The others there has been nothing for the past couple of months. So if anyone has any comments on any of those, and if they believe that any of those topics should be revived or we should just kill them off...
AUDIENCE SPEAKER: Hi, Gert certificates key, author of at least one of those. Just contact the authors and ask if they are still interested, because if there was no feedback, support, or the other way, whatever kind of comments for a number of months, that's probably just a crazy idea of the authors. Nothing more.
CHAIR: Thank you.
AUDIENCE SPEAKER: Nick Hilliard, INEX, I think there is a lot of legitimacy in a lot of these proposals and I would hate to see if they were just abandoned simply because people have too many other things on their plate. So, can we try to bring them back to the mailing list and get some work done on them. We are a Working Group and I think they have legitimacy.
CHAIR: Do you think for example if we bring them back one at a time not like the whole list and we try to generate discussion, focus on one ‑‑
NICK HILLIARD: I think that would be a better idea. 7 is app handful. One is manageable F you present 7 at the one time no one is going to do anything, because they'll assume that somebody else is going to do something. It's just not going to happen.
CHAIR: Okay. Thank you very much.
AUDIENCE SPEAKER: Michael from the RIPE NCC. I have Sebastian nice I think err on chat from nor I say network. He says be, as the one having proposed numbered work item number 7 I must say there seems to be rough consensus for just adding an Abuse‑C field to relevant objects. This is not abandoned and I would like it to continue.
CHAIR: Perfect. Thank you very much. Thanks for the feedback.
Okay. Then this will lead us to the next section. I think it's the presentation from Denis who can not be here.
If some of you knows, Denis is the new co‑Working Group Chair and unfortunately he had a heart attack a couple of weeks ago so he could not join us here today. He made a presentation with a voice recording to present his views on the database.
He is doing fine. He is just not recommended to travel at the moment but he is doing absolutely fine otherwise.
DENNIS WALKER: (Voice recording):
This presentation is about solving the unsolvable. What do we mean by that?
Over the last few years many database issues have been brought up, but they have stalled. A system was introduced for numbered work items, the idea of that was to try and move these items forward, and actually achieve a solution. We currently have 7 of these work items. 5 of them are still at the problem definition stage. They haven't even got past the first post. 4 of them have less than 5 comments and only from at most 3 people.
Some of the other issues that have been raised over the years include locking of over 500,000 personal data items in the database by the RIPE NCC. When they did this, they asked the community to (700,000) the community to give them some guidance on what to do next. This was only supposed to be a temporary arrangement. But no guidance was begin. Implementing UTF 8 in the database. Technically this was done a few years ago, but nobody moved on and considered the political or the operational issues with actually having non Latin one characters in the database.
To password or not? This is one of those issues that comes up every couple of years when people ask should we allow passwords to access the database? Or maybe not. Should we force everybody to use single sign on methods? But no one ever makes a decision on these things.
Validation of contact data, should this be done at least once? Should it be done repeatedly over a period of time, or should the RIPE NCC never validate contact data because it's not their job? Again, this comes up every couple of years and people ask questions. But no one ever makes a decision.
Personalised authorisation. This was an issue I brought up quite a few years ago. There have been at least two proposals done on it. But it hasn't happened.
Public versus private data in the database. There are those who argue that all data in the database should be public. But we broke that model when we hid the password hashes. Some would argue that a lot more data should be private, but what do you mean by private and how private and how much data should be private, if at all?
Is stale data still valid? If some data hasn't changed in 15 years, do we even know the resource holder is still alive? Do we need any kind of system to touch these objects to show that they are still in use?
Many of these issues are often brought up just before a RIPE meeting. There is some strong views expressed during the meeting. But then they are quickly forgotten straight after the meeting. They never resolve these issues and they are brought up again and again over time.
Even what seemed to be relatively straightforward issues never get concluded either. For example, continued use of the database as a white pages directory. Should it be allowed or not? There is only half a dozen people actually use it anyway. Should we continue the use of the geolocation language features? These were added as a kind of experiment, some people like them, quite a few people didn't like it, they thought it was the wrong place to put it, but the argument just rumbles
On and never goes anywhere.
And these are just lists I put together from memory. If I actually went back through the mailing list archives I'm sure we would find quite a few other issues that were brought up and again have just stalled.
So, what can we do now about these issues? What do you do when you can't see the way forward in a problem? And the answer really is quite simple. You move.
Where do you move to? You move to a new position.
It doesn't matter what direction you move in, it doesn't matter how far you move, just move. Take a giant leap, take a small step, it doesn't matter, just move the situation on to a new position.
From that new position, you might see the way ahead, if that step you took wasn't the complete solution, where you are now, having moved a little bit from where you were, you might see how to solve the problem. But if you stay where you are, all you are going to do is continue to not see the way ahead and that seems to be the where we are with many of these issues.
So, ultimately there is only three ways to solve a problem. You have three options:
You can either change something or replace something or fix something. Maybe that offers a complete solution to the problem, or maybe it just takes you a little step along the way into a new position. But at least you have done something.
Or you can accept that it is a problem and live with it. Maybe it's not that important, maybe you consider that it's not worth the time and effort to actually solve the problem, which is fine, because that is actually a solution. It's a conclusion, as long as you document it so that if the problem is brought up again in a year or two and nothing has actually changed, you can point to the documented decision that was made about it.
Or maybe you decide it's no longer a problem, or maybe it never was a problem and you drop it. And, again, that is a conclusion. You have made a decision and you document it so if somebody brings it up again, you can say well last time we talked about it we didn't consider it was a problem, and refer to it.
So what is the way ahead? As I see it, we take each of these issues one at a time. We discuss it and hopefully we'll get more than two people making a comment. Now I know many times people look at somebody else's comment and think to themselves yeah, that's what I would have said and then they don't say anything because they think their point has already been made. But if we only get two people or three people actually saying anything, it's very hard to decide if that is a consensus.
So, hopefully if a few people join the discussion, we'll be able to make a decision. And having made a decision, we can implement that decision, even if the decision is just to document the fact that you are not going to do anything. At least that is a decision. And then you can move on with life.
So, any questions?
CHAIR: No questions for Denis. Okay. Then the next item... Greg.
AUDIENCE SPEAKER: Brian Nisbet from HEAnet. I don't have a question. I'm just wondering what comes next. We just talking about doing something, a whole talk about doing something. And we are ‑‑ I am just wondering if we are going to do something or whether that is going to disappear into the eater like a lot of the other things that we are talking about (ether) so, this is ‑‑ it's a lovely thought exercise but if we all just sit here and go okay, cool, then we're going to be in the same situation we're in and maybe we're okay with that but we should document that as Denis says.
CHAIR: Denis is actually available on Skype I believe to answer any questions.
RANDY BUSH: My observation, watching this for a little while, is that when something needs doing, we are deciding it gets done. When something really doesn't need doing or it's an interesting idea, it gets futsed around with for two or three meetings and nothing happens. This is a feature not a bug. It's okay. Right. If somebody has a bright idea and we discuss it for two meetings and we leave it on the floor, it's more polite than documenting you're an idiot
RUDIGER VOLK: I certainly agree with Randy to a large extent, but let me throw in a few additional comments. I quite clearly object to the idea that well, okay, if you feel the itch you should be moving even if you are in really thick fog and actually moving in the thick fog when you do not see the toes of your feet, well, okay, you might actually travel much further when you are next to Anna BIS than you intended. (Abyss) and yes, I guess that a couple of items on your list were where you note that well, okay, nothing has happened. Yes, people have made ‑‑ having proposed that there are certain problems and they would suggest to move in some ways and in many cases there have been strongly voiced objectioning to it, and as Randy says, being polite, we have not documented that really. And I'm kind of not liking the idea that we really have to put effort into that kind of terminating a project that does not really start. Though, of course, if the idea comes up every two years, it would be really nice if we could point to yes we discussed that two years ago and four years ago and six years ago, and either all the time the need for doing that proposal was not really there, or we would have done something according to Randy, or yes, there was an argument that well, okay, don't do it.
DENNIS WALKER: One of the issues that I would mention in that respect is out of region and route objects, this came up for four right meetings in a row where there was a discussion on it just before the meeting there was a proposal on it just before the meeting, there was proposals on how to solve it and it was dropped straight after the meeting. That's the kind of issue I'm talking about where is it a problem or if it's not a problem? If it's not a problem let's just admit it and write it off. But if it's still considered to be a problem, let's fix it.
RUDIGER VOLK: Kind of the question there it what it actually a problem and what is a good solution for it? There are many ways we can make things worse. In my experience, actually the good ideas and the good moves are scarce and there are infinite versions that are worse ‑‑ well, okay, kind of for everyday talk it's infinite, and that particular discussion I think we actually did not get to good solutions. And so ‑‑ so the result is what one would expect in that way.
DENNIS WALKER: I would disagree there. I think there were some good solutions on the table but nobody made a decision.
RANDY BUSH: That's because there were a significant number of people that didn't think they were good solutions. And when that happens, we Wedge and this is a feature not a bug. In particular, this problem has some interesting hidden agendas that aren't really that well discussed, for instance those people who want to make it region only, believe that they can purify the data in some way by getting those nasty others out of here. And a lot of people don't buy into that movie. And I believe it's wedged because there are no clear good moves. And that's okay. And I think... well I have seen over the years here is when the good moves appear, people go dam, okay recollect let's do this and we do.
DENNIS WALKER: But while we have those objects in the database, we can make things a little bit better.
BRIAN NISBET: So, without necessarily commenting on the specifics of anything, and I mean hey I'd love to see, you know, the Abuse‑C stuff pushed forward and I'd love ‑‑ there is a bunch of things there the but it's what Ruediger says and I think it's the conversations that we have, especially with external groups in regards to decisions that haven't been made. The decisions that have been made it's grand, it's clear, but there are decisions that haven't been made and we sit there and as you say they crop up every two years or so and it's actually quite difficult. It's gather around the fire children and we'll tell you about the time that we discussed this particular change, you know. Grandad Brian or Ruediger or Randy is talking. And it's not clear and we deal with the good folks in law enforcement, we deal with the good folks in governments and some of them are like why haven't you done this thing, it's like well... one upon a time in a meeting room far far away... so I think there is something we could do, at the very least there is something we could do there around if not being impolite and, I'm all for politeness, but it goes to a point where we're causing problems for ourselves and we are too polite. We could, at least, go here is where this got to, here is a clearly documented point that we can say, and that's fine we don't have to be impolite and say the community did not reach consensus for these documented reasons. We don't have to tell anybody that it was a terrible idea. Or at least not use those words but we could kind of get to that point and maybe with some of the other ones, I'm possibly slightly less risk averse while being aware of the fact that I'm not the greatest expert on the RIPE database, we could try some things now and again. But you know, I'm not saying we should do everything just to do things.
RANDY BUSH: Brian, thank you for volunteering to write up the resolution of this one. Set the example. And I really mean that. I don't mean don't do it. I think that's a good idea. I for one couldn't document it in a way that I think would be appropriate.
BRIAN NISBET: I agree. I'm not ‑‑ I suppose even if ‑‑ I'm not going to try and think on my feet here. But if there is something minuted, maybe there is something there we can just go here is a point we can refer back to. I will undertake to certainly spend some cycles seeing if I can think of a way that this might be doable in a sensible fashion and come back ‑‑ there should be stuff there, but I'll have a think about it and see. I will undertake to do that if the Working Group would like me to.
CHAIR: Thank you very much. And thank you Denis.
DENNIS WALKER: Thank you.
CHAIR: And I think we will move onto the next item. So, Greg?
GREG MOUNIER: Good afternoon, that was a very, very interesting discussion, because what I'm going to tell you now is basically yet again a new proposal to do something great, but I hope you tell me before, don't leave me hanging for five years before you tell me no that's a bad idea, just move on and do something else.
I was invited by Brian. I presented last year at the anti‑abuse Working Group. By the way I am Gegory money yeah, I'm working for the European law enforcement agency in particular the cyber division. I presented already in the Copenhagen in the anti‑abuse Working Group and in Madrid in the Plenary about the problem that the public safety agencies are investigating crime have sometimes used RIPE or the APNIC or the ARIN or database and it's related to WHOIS accuracy. And I was asked to just give you a brief update on what I have been doing recently and just also to engage with a new Working Group.
Really briefly. The problem statement is that WHOIS accuracy is very important for many different things. Of course the first purpose is trouble shooting for you and for the technical community. But you have a lot of other actors that are also using the RIPE database to try to attribute malicious online activity investigations. It's not the only means, but that's one of our most important means. Actually the first port of call when you do an investigation and you investigate a malicious IP the first thing you do as an investigator you go and do a look‑up in the WHOIS database. And RIPE NCC and the RIPE community has done a lot of efforts to try to improve the accuracy requirements. You have a number of policing and contractual requirements to keep and maintain information, registration information accurate in the database. You have also implement add number of ex anti and ex post control and that's all great and still, from the evidence, at least based on my community, the law enforcement and security agencies, we do see that WHOIS accuracy is still a bit of a challenge.
Why is it? So we have tried to think about with our colleagues, had discussion with a lot of you, a lot of members of your communities, we also talked with the RIPE NCC people and the rest, and I think we have tried to identify, we have identified at least two reasons for that. One reason is that from our perspective the database does not properly reflect the entire chain of assignments and suballocations. So when RIPE allocate a block of IP space to an LIR, then it's fine the registration information is usually correct and accurate and up to date. But then when LIRs assign it to clients and then clients or assign it or suballocate to clients further down the stream then either it's not recorded properly in the database or it's not just up to date and when you do investigate and malicious IP it's not an IP held by a big established ISP of course, it's most of the time block of IPs that have been sub allocated several times to a small ISP and the information in the database is not up to date and accurate, purposely or not purposely. I don't want to comment on this.
The other aspect we have seen is that there is a lack of compliance mechanism in that sense that to ensure accuracy requirements are also implemented by down stream LIRs. I have been through the policies of RIPE and it seems to me that there is at least a policy obligations for the LIRs to, they are kind of responsible, so to ensure that the registration information from their clients, once a block of IPs have been assigned to down stream providers, have to be maintained up to date. But there is no compliance mechanism. So I think that's also one of the weaknesses of this system.
In Madrid I presented a case study whereby it was based on a real case where investigators were investigating somebody who entered into the network and stole a lot of customer informations and then they could trace, on the basis of forensic investigation, they could trace the attack on one IP and the first thing they did was to try and identify the address of the latest down stream provider to that IP so they could serve to try to find the subscriber of that IP address. They contact a lot of Open Source research in order to identify the address of the hosting provider. Where do you send your legal order? As an investigator you need to work with an investigative judge or prosecutor. You provide evidence. So you'll see this is the IP logs, here is the malicious IP. Please give me the paperwork so that I can seize the server or at least speak to the hosting provider or the ISPs.
And then I went through several slides, and at the end of the day after several hours of research, we ended up with three different addresses in the UK, one address in Serbia, bun address in bell ease. One US number, one Swedish number, one UK phone number. What do you do, you go whack to prosecutor and say we can send four M laths to different addresses in the real world you are can't do that. Either you drop the case or you find other ways.
Just quickly, for the contractual requirements. Forth RIPE NCC members, it's very clear. The RIPE NCC standard service agreement says that members are required to maintain correct registration data. And then you have got article 6.4, that provides compliance issues. So if you do not comply then suspensions or deregistration. That's pretty straightforward. When it comes to suballocation, the IPv4 address allocations assignment policy say that all assignments and allocations must be registered. And registration at that time must be maintained at all times. Then if you move on to section 5.4 you see that the LIRs are contractually responsible for eninsuring the address space allocated to it is used in accordance with the RIPE community's policies. Which include accuracy requirement.
But there is no requirement mechanism. So, what we need from the public safety community is that registration of all IP assignments and suballocation providers to downstream providers would be registered in the database so that the entire of suballocations issen accurately reflected flexed. We don't want to the database goes to the end users, this is not what we are after because we need a proper address for a company to serve a legal order. And we want ways to ensure that there is adherence to the policy requirement.
Issues to be addressed. Again this is just a couple of months brainstorming with some you. We have identified three main problems I think. First, compliance with existing contractual policy obligations. How do you devise a compliance mechanism which is not too expensive, really easy implementable and that the community is happy with. You have different options. For instance you could think of a centralised option where you could ask for instance the RIPE accuracy compliance programme to be extended not only to the members of the RIPE community but also to the local Internet registries and the customers and the customers and the customers. We have done a bit of calculations of what it would cost and so on and I think probably realistic to implement, that would be too expensive and not properly implementable. Maybe there is a second option.
Could you think about a decentralised system where by assignments and suballocations would be dependent, made dependent on the existence of a compliance functions with each LIRs or small ISPs. So even if you are one man show in your company, then you would have the responsibility to ensure that your registration information would be up to date, and expert due diligence and expert control would be done at the closest level. So the responsibility doesn't lie with the RIPE NCC or with the RIPE members, but it lies with the proper ISPs or the one just before providing the connectivity.
And the another issue is can the RIPE database technically reflect more than one level? It's not completely clear to me. But I got messages for instance Marco raised it that in fact the RIPE database might not be able to go more than one or two levels to you have the allocations then the assignments and then technically you can't reflect the suballocations. I don't know but that's what I was told so maybe we could look into it.
And then is it possible to allow for more level of assignments? And then what needs to go for instance to the country attribute? Is it the physical address of a company or is it the administrative address? I give you an example. Very recently we have had maybe you have heard about the avalanche operation which was a massive criminal infrastructure which was taken down by Europol, in fact in 30 different jurisdictions and seven of the back end servers, we had trouble to identify where they were because in the RIPE database and other RIRs databases they were all registered in bell ease. Of course we knew from experience that the servers on the RIPE were not actually in bell ease but what do you do when you go and see your investigative judge and say that the RIPE database says apparently it's registered in bell ease so you should send your MLat to bell ease. At the end of the day browsing table and the rest you identify it's probably in Romania so you do it, but legally this is a little bit on the fringe because why would the judge deliver an MLat if the RIPE database said it is in bell ease. So that's the type of issues we have. So maybe just defining what it means to country attribute because it's perfectly legal to have a country registered in bale ease while the server is based in Romania.
Last slide. Way forward. We are still in a process of brainstorming with interested stakeholders, we are trying to collaborate with you to find and industrial led solution, something that doesn't minuted err your processes too much. It's practical and implementable. Trying to also think whether we need a policy for that because it's touching many different issues. Do we want one policy change and then we solve all the issues? I doubt it. And really start the discussion on the mailing list, maybe on the Database Working Group. That's what I understood from Brian and co‑chair of this Working Group. Then hopefully, present something concrete at the RIPE meeting.
So any contributions, suggestions? Thank you very much. And I hope you will not let me hang for five years. I have got other things to do. Thank you.
(Applause)
RUDIGER VOLK: Greg, one kind of question I have is ‑‑ well, I think it will revolve in the series of questions. I guess you have worked with the RIPE NCC extensively to get an understanding what is available in the database.
GREG MOUNIER: Do you want me to respond now?
RUDIGER VOLK: Yes or no or ‑‑
GREG MOUNIER: Extensively not as much as I would like to, but yes we are in constant dialogue and in particular with Rumy we are establishing the series of webinar to try to increase the knowledge of how the RIPE database work for all the investigators so that they can find the information they want in the database
RUDIGER VOLK: My question would be, or request to you would be, can you give us your identification which pieces, which attributes and objects in the database you actually require with what understanding of what you think there is, like say my understanding of administrative and technical contacts is not really very current because the last time I really was thinking about it was something like 25 years ago when I looked at the documentation that was used with SRI NIC and today I think kind of I would not know something that actually and precisely defines what's expected there. But you should have very precise understanding of your requirements.
GREG MOUNIER: Shortly now, if you ask me now I think it's really an address for a registered company, the condition which is providing the last connections to the IP. So, if Liz web is providing a block of IP to inif he were owe who is providing the block of IP to another company I'm not interested in Liz web or infern owe, I am interested in the last company who provided that because Liz web may be in Germany but the other company is somewhere else. As an investigator I need to know in what country and what national authorities have to send my mutual legal assistance request
RUDIGER VOLK: Okay. Kind of I'm commenting two ways, one is okay, actually what's in the WHOIS and what the hijack err actually of that address is. Of course are two different things and you should actually be going after the hijack err. Admittedly probably only of a low percentage of attacks actually happen on hijack, but nevertheless...
PETER KOCH: Greg, at the risk of you having a day gentleman view somehow because part of this was somehow mentioned in a meeting the last time or before that actually. So, my understanding is that to actually access the database and do your investigations, there is like a Trinity of say legal grounds that you need. One is more the law enforcement officer to actually be legally able to use it and we get the point that they have that by their own, in their own jurisdiction and so on. They can access that as a civil servant and so on and so forth.
The second one is and it was clarified last time, by using it for the purpose you or your colleagues use it, not violating the terms and conditions, and that was clarified by Athina, so, also from that provisioning and consumption side the database may actually be used for it.
Now the third part is the purpose of the database. And that is document on the RIPE NCC, or RIPE database website I should say in the RIPE database documentation 2.1.1, and I'm not going to bore everyone by reading this down here, but law enforcement needs are not explicitly mentioned there. The one that comes closest to your requirement is to provide accurate registration information and (we heard in the previous session that the definition of accurate might be different in different communities) to provide accurate registration information of these resources, that is INET NUMS and everything else in order to meet a variety of operational requirements. Now we can dance on the needle whether that's operational requirements or not at the time this was written operational requirements were definitely network operational requirements. So I'm trying to say here is what you are asking for might be a bit bigger than you think and that some people in the room think. This is adding a purpose to the database that hasn't been codified to date. So this is a bigger thing, so, nobody is going to hang you for four years or let you hang, sorry, for four years, but it's ‑‑ it's definitely a repurposing or widening the purpose and that needs to bit more than just doing things here and there. Thank you.
(Applause)
RANDY BUSH: Further to Peter's comment, the purpose of the database is for our use. You are welcome to use it. But asking us to change it for your use, and it would be a serious change, will seriously affect our community. And as you found out that those servers really weren't in bell ease, they were in Romania or wherever the hell they were, that's your job, and Geoff Schiller once said very well that the law enforcement is not easy. When it's easy it's called a police state.
GREG MOUNIER: So basically you are making my work more difficult so that we remain in a rule of law as a society. That's interesting.
RANDY BUSH: Not making it more difficult. But going very cautiously about making our work more difficult to make yours easier.
GREG MOUNIER: I appreciate that. That's also something ‑‑
RANDY BUSH: And we sit here just for our operational use, the current definition of what it's supposed to be for, we have real problems making it accurate. Making it accurate for your use too is a much further step and much more difficult. I mean, you know the problem you have using your data to accurately find something. Asking a bunch of hippy operators to do it, who are you kidding?
GREG MOUNIER: Well maybe I was also hoping that maybe our interests would be in line with some of your community members and that some of you would have also an interest to have a database which is a little bit more accurate in down stream. But ‑‑ if it's not the case then that's fine and we move on.
NICK HILLIARD: I think there is an issue that you haven't really brought up, and that is using the RIPE database as a mechanism for obfuscating contact details on an intentional basis. Now, a lot of people looking at this sort of thing have seen a lot of registrations for address space which is obviously used in some areas in the RIPE NCC service region which is registered to legal entities outside the NCC service region in jurisdictions which are well known for facilitating privacy requirements, one of those would be bell ease I don't see any easy way of fixing this problem and this is an extraordinarily difficult problem to fix. If you have suggestions, concrete suggestions which could be useful in this area, that's fine the but coming here to sort of say well look, somebody has an IP address block registered in Serbia or used in Serbia, or with a contact details in Serbia, bell ease and Romania and the UK and everything like that. That's fine, we're aware of this problem but don't necessarily know how to solve it because within the terms and conditions of the registration information that have to be provided to the RIPE NCC, this is strictly and technical correct. But really what's happening is that the end user or intermediates are using this technical correctness in order to hide themselves. So, please, do make some comments.
The second issue that I wanted to bring up was the issue of PA address space. And what to do about LIRs who don't maintain their PA address space ACK recommend. And this is an enormous problem has been with us since the beginning of the RIPE database. I don't really see any way of fixing this because, to be blunt, if they don't maintain the information accurately, there is very little that the RIPE community or the RIPE NCC can do to beat them into line. I mean I guess you can threaten them with non registration or deregistration or some other sanction, but I don't think that's really going to fix a problem. So, again, if you have solutions, we're willing to listen. But these are ‑‑ you are bringing up long term problems which we are aware of and for which we don't necessary have good solutions ourselves after a lot of years of looking at the issues.
BRIAN NISBET: So, as Greg says, we have discussed this and I kind of have a couple of points. I suspect I have a different point of view to this to a number of people who have spoken already. One of the tones of kind of a lot of what has just been said is very much of an us and them kind of feeling, which I don't like personally. That's my personal view on things. We have had an increasingly good relationship, in my opinion, with the law enforcement community and while I do not wish to live in a police state or otherwise very vehemently do not wish to live in that, I feel that. The Internet has got to a point where the RIPE community cannot just be operators. And we're going well beyond the scope of the database Working Group at this point in time and I fully accept that. But I don't think we can say this source of information is for us and not for other group, obviously in this case, law enforcement, where it is vital where it is the only information source, and Randy you say ‑‑ okay, it's the authoritative information source, I mean you say it's law enforcement's job to figure out that the server is somewhere else. Yes of course, detectives have got to detect, that's why they get paid whatever non big bucks they get paid. But I think it is up to us as the people who operate these networks and operate these communities insofar as as is possible and in so as far as works is we should be working together as a to improve this resource, this Internet we have built which has gone well beyond the dreams of 25 years ago, to work for the citizens of the world. Very highfalutin notion, etc., etc., but in my conversations with Greg this week, we knew we weren't coming to this room with a solution. And I think that the scope of the problem is massive, and there is no question there.
But we're the experts who have to work with law enforcement to try and do this. It's not saying to Greg come back to us if you have a solution. It's, okay, can we work, can more of us work in this way? And I know a number of people are, to trynd an improve this, to try and look at what those steps are, not one big sudden solution. But the steps might be towards something that is better while keeping in mind the technical constraints on our side and the needs on their side. And reach insofar as is possible some manner or class of compromise.
RANDY BUSH: First of all, people should note that the stenographer not only got SRI NIC, but got highfalutin, I mean come on...
STENOGRAPHER: I am Irish!
RANDY BUSH: Nick, don't worry about the British database because they are going to be cut off and float away. We can't make the database be accurate to a dam e‑mail address to which we can get an answer. Dreaming that we're going to solve your problem is. But, let me give you something optimistic. In my day job I'm actually a researcher, a measurement researcher, and if somewhere in your culture of law enforcement, you haven't got people that haven't run traceroutes to everywhere and mapped the Internet down so that you know where that bloody address actually is in where the heck it was, Romania. You are a bunch of fools and I'm sure you're not. Right. So you have already got something much more accurate than our database. In fact, I would ask you to help us update it. Seriously. Seriously. You have got ‑‑ you can develop much better data than we have got. What we have got really sucks.
GREG MOUNIER: Just briefly ‑‑
RANDY BUSH: If it was good we wouldn't have this Working Group.
GREG MOUNIER: I mean, it's all super good input anyway but because I have spoken to a number of you and yes you came back several times with the fact that the RIPE database might not be the right tools for the purpose we have. But two points.
First of all we only, you can't expect ‑‑ you guys are experts in BGP routing, in traceroutes, you can't expect grass root investigators that are not only investigating cybercrime but any type of crime where an IP address is involved, which is 80% of the crimes now ace days to be experts in traceroutes and BGP routing, otherwise they wouldn't be cops and they would be attending RIPE meetings and devising database. So from that perspective, we can increase the knowledge and the expertise of the investigators, the grass root investigator to certain level, but don't expect them to be using super‑duper techniques and instrument to trace an IP address. I mean we are law enforcement. We are not intelligence services. We are working in a very tight legal framework. We don't get paid a lot and he we just do the job we can, with the very limited input we have and it happens that there is a database, there are five databases where you can find theoretically, the registered address of a company who provides and who holds the resources. That's just a fact. So, the fact that we are engaging with you to see if there is an interest in maybe increasing the accuracy of that database, I think it's not ‑‑ you know, it's not crazy demand. I get your point absolutely.
AUDIENCE SPEAKER: Carlos from FCCN, the Portugese NREN and also the general assembly Chair for the national CSIRT network in Portugal. Well, first thing, thank you for your presentation and it's always good to see you in this forum. There are several stuff that I want to comment. I usually get to see things in the database that I don't like. I'm not reporting all of them or most of them. I also don't have a solution. But there are tools, for instance if you go to stat dot RIPE database.net well that helps a lot. So ‑‑ and anyone can access it. So you don't need to be an Internet wizard or a very experienced network citizen to use it. And there is a lot of data.
I was a bit puzzled about Randy's comment about the RIPE database not being authoritative. I'm not sure if I understand that. Yes, it's not accurate we all know, and Gegory already knows a lot about it. So, what can we do? We can keep the channel open and we need to address this and try to address this. I'm not sure if it's possible ‑‑ well if an inaccurate record is detected, we can report it and someone is going to do something about it or not. I'm not sure if it's possible for the RIPE NCC to get any mechanism that can raise flags if the data which is seen from stat.ripe.net doesn't make any sense when you cross it with something which is in the RIPE database. So that might be a small step. I don't know. And that's it. So thank you again.
RANDY BUSH: On the hypothesis that you have within your social structure, a pretty good database based on actual mapping techniques, then you have a serious advertising problem if law enforcement agencies out in the Boonies now how to use WHOIS more than they know how to reach that. Was that an unclear statement? In other words, I am a law officer in Montana, I am going to use WHOIS or I'm going to use the law enforcement cultures, you know, you should be advertising your database better internally to the law enforcement community. Because, finding the WHOIS to a normal human being is a little obscure.
GREG MOUNIER: But it's not necessarily for investigators. It's not necessarily obscure for investigators.
RANDY BUSH: If it's less obscure, then an accurate, more accurate database you have, then the people running that database are not being very good at advertising it within your community.
CHAIR: We need to cut it sort unfortunately.
WILLIAM SYLVESTER: All right. We have about three minutes left. Anyone have any other business?
Piotr: Just a short question, there was a habit for that Working Group that the agenda was going to be sent well in advance like at least a few days. I checked the archives and there was no mail with the agenda for the meeting. Why?
WILLIAM SYLVESTER: I don't know.
AUDIENCE SPEAKER: Okay. Fair enough.
WILLIAM SYLVESTER: Anyone else? Great. Well thank you so much. We'll look forward to seeing everybody in Dubai and some of these discussions today were great. I'd encourage you to take discussions off line. Feel free to reach out any time and let's take some of the these discussions on the mailing list. We'll see you guys in Dubai.
(Applause)
(Coffee break)
LIVE CAPTIONING BY
MARY McKEON, RMR, CRR, CBC
DUBLIN, IRELAND.
