RIPE 75 ‑‑ Dubai, UAE
22 October 2017
Plenary session
2 p.m.
HANS PETTER HOLEN: So, welcome to RIPE 75 here in Dubai. It's an honour to be here and thank you very much to your excellency for our local host from TRA and from the local connectivity sponsor, and welcome everybody to this really nice and warm place to have a RIPE meeting in. As, you know, my name is Hans Petter Holen, I'm RIPE Chair, and I am from Norway, and when I left to come here it was actually freezing on the grass outside my house.
So, during the newcomers' session, the fire alarm went off every five minutes, so I guess this is just to test my abilities to handle stress today.
We have, so far, 526 registered attendees and 272 has checked in as of 13:30 today, so this is looking good to be a great meeting so thank you very much for coming. The RIPE meeting is a meeting, it's not a conference. It's open to everyone and what we want to do is to bring people together from different backgrounds, cultures, nationalities, beliefs and genders, and we want to be a safe and supportive and respectful environment and I think it's particularly important to the community that we are actually going outside the EU, the European, that we are coming to the Arab world that we are going further east than we have done in the past because there is a large part of our community that is here and it's really good to see so many newcomers from this part of the region.
We have some code of conduct. It's really be nice to each other, be respectful and in the event you experience something that is not good, we have trusted contacts, it's Mirjam, it's Nick and it's Rob, and if you feel the need to speak to somebody, because this is a large group of people, they are here to help new case you need any assistance.
We have a Working Group working on increasing the diversity. So, there was a group setting up this at RIPE 73. We will have a lightning talk today on the progress on this. There will be a BoF on this tonight and a task force meeting tomorrow and there will also be a women in tech lunch on Monday, and there are some really good young women speakers from the region here, so, I'm really looking forward to hear what they have to share with us.
And this is a topic that's especially important to me, because I have two young daughters who are actually coming into this industry as well, so I really want to see that we are open and receiving all members of this community into our events.
We also have with been working on accountability. As you know there has been a lot of talk about accountability in the ICANN world and in the Internet governance world in general, there has been a lot of good work done on documenting the accountability of the RIPE NCC. What this Task Force is working on is to accountability of RIPE. Because as, you know, there is the distinction between RIPE as the community and RIPE NCC as the membership association. And RIPE as the community is not sort of formally designed. It's got a lot of documents and it's got a lot of tradition, and this, the Task Force is working to analyse that and see if there are any gaps that needs to be better documented moving forward. So far it's working really good and I'm looking forward to see the report from the Task Force at the Plenary tomorrow at 4 o'clock.
So, one of the things that is also new at this meeting is Internet of intention and I tried to figure out what is this Internet of things doing and apart from all the gadgets I have in my network that I have had there for years, there was this ad going to the Norwegian newspapers where one of the operators said we are now launching the next generation IOT network and here is what we are doing. We are connecting the sheep. So, if you know a bit about farming in Norway, it's a lot of hills, and we have a long border with Sweden and in Sweden they have wolves and the wolves don't see the border so they come across to eat Norwegian sheep. The idea is to actually have the sheep connected to the Internet so that the farmers can know where they are and probably protect them from the wolves. I don't know.
Moving a bit deeper in my research. The other extreme is kind of the oil companies, like Statoil, they are building large sensor networks on the sea‑bottom in order to automate much more of the oil products. It's a mind‑blowing different way of things about things where security has a completely different dimension than what we think about.
If you look on the RIPE website you will see RIPE class probes which was a project which didn't think as Internet of things when we started but it's actually a lot of probes, more than 10,000, which is connected to the Internet and there is a very good article there written about some of the design principals and how to keep and manage such a long sensor network in a secure way. If you are interested in this, you can show up in the IOT session on Tuesday.
We also have a foundation set up and I talked about this at the last meeting, so Rob Blokzijl was our previous Chair and in memory of him, we have formed a foundation in his name that will hand out rewards to recognise people in our region. So the board of the foundation is me as Chair, Nigel Titley as treasurer and Daniel Karrenberg as secretary. And the next step in setting up this now is that we are looking for an awards committee that will decide who will get the first award. So if any of you have any input to give to us on who should be part of that awards committee, please approach us before we move ahead on this.
You all know the meeting plan by now. You have it in your paper‑based app around your neck, you can see the plan there. And as you can see, there is a full week with excellent work and excellent presentations coming up. And the work behind this is carried out by the Working Groups, as you can see on some of the days, where the Working Group Chairs is doing much of the work. You can see there is an open spot here, so one of the Working Group chairs in the Database Working Group has stepped down so if any of are you interested in this, you can go to the Database Working Group and volunteer.
It's important that you look at this as a meeting, not a conference. So, we all encourage everybody to participate, go to the microphones, ask questions to the presenters, in discussions. Since we have many remote participants, please state your name and affiliation so that it can be clearly recorded by the stenographers and the scribes. And it's really about making this meeting great together.
The programme for the Plenary Sessions is put together by the Programme Committee, and Benno, as the Chair will speak a bit later on the work of the Programme Committee. I would like to point especially to Khalid, who is the representative from the local host, and he happens to be the Chair of the MENOG Programme Committee and then Osama who is the representative from the MENOG committee. And then we have representatives from the other regions like ENOG and SEE and the remaining participants are elected by the community and there will be an election later this week as Benno will talk about later.
So, I told you about the paper‑based app. There is also a digital app for your mobile phones. It's made by the RIPE NCC, and it's tied to your RIPE NCC access account and you can use that to network and set up with other participants in the meeting. Download it and try it out and give feedback to the RIPE NCC because this is something they have developed based on the requirements that have come up over the last couple of meetings and it's important that we get feedback in order to further improve it.
The RIPE meeting is not only about the presentations and the Plenaries. It's also about networking, human interactions between the sessions in the coffee breaks or at the socials. So, we will welcome you all to meet the RIPE NCC Executive Board at 18:30 today at the 6th level next to the pool and there is a welcome reception after that. And then on Monday, there is a networking here and on Wednesday there is the traditional RIPE dinner and for the RIPE dinner you need a ticket but for the other events you can just come with your meeting badge.
Then I would like to thank our sponsors, that you can see here. Without the sponsors, the meeting wouldn't happen. But I would also like to extend a special thanks to the local host for TRA and I would like to introduce His Excellency, H.E. Harmad Obaid Al Mansoori, and welcome you to the stage to give your opening speech.
(Applause)
H.E. HARMAD OBAID AL MANSOORI: Your Excellency, Hans Petter Holen, Chairman of RIPE, Your Excellency, Axel Pawlik, managing director of RIPE; CEO of Emirates.du, number of RIPE Executive Board, ladies and gentlemen, it gives me great pleasure to welcome you in the United Arab Emirates here in Dubai and to address this five‑day event where hundreds of Internet service providers, network operators and interested parties gather to discuss issues of interest to the Internet community. It also gives me pride that the UAE is hosting the only RIPE NCC office outside Europe. I see this as an information of the UAE regional position as well as its prospects for growth and development in the future.
We, in the United Arab Emirates, have adopted the Internet as a driver for prosperity since the early stages. It was in 1999, when our leadership launched the Dubai Internet city with some people at that time considered as a leap towards the unknown. Today, the Internet penetration in the UAE is over 90%, making it one of the leading countries in the world. The UAE hosting IX, thanks, the regional biggest Internet traffic exchange platform that corrects global network and network provider and content provider in the region. In terms of the service index, the UAE is ranked 9th worldwide as UN survey for the world development and we are targeting to be position number one in 2021. A few days ago, our leadership announced the UAE strategy for artificial intelligence, and by the way, the government announced Minister of artificial intelligence before two days.
The first of its kind in the region with the aim to transfer the country working environment through investment in advanced technologies and artificial intelligence. This strategy comes as a first step towards the realisation of the UAE future vision to improve government and private sector performance.
While we are rapidly advancing towards the future, we come more aware of our needs for robust connectivity to accommodate for that exchange among millions of devices, this will require more and more Internet protocols, that's why we are key network transition towards IPv6, a journey that we already began, the TRA is leading the UAE effort to shift to IPv6, a transition that reflects the expansion of the Internet address directory in the country, and, by the way, we are using IPv6 in this hall.
In all those efforts and plan, we are putting RIPE. The organisation that have proven to be strong support for progress and modernisation. I am confident that all countries that fall under the RIPE responsibility acknowledge the importance of the role played by this organisation through its career. One target of the 9th global within the UN sustainable goal is to increase access to information and communication technology, and starting to provide universal and affordable access to the Internet and in least developed area by 2020. Here where the real role of RIPE lies. Your Internet has clearly embarked the growth and development in all countries, and especially the ‑‑ your peering plan, contribute to reduce the cost of Internet operation which contribute to ease the economic burden on countries enabling them to use the saving in other areas of development.
I, therefore, take this opportunity to thank RIPE, wishing all success for this organisation in coming years. Thank you.
(Applause)
Ladies and gentlemen, unfortunately the wide horizon opened by this future technology development do not come without side effects. You may agree that cyber security is, and will be, a real issue visited by Internet today and tomorrow. In the area of Internet of things, anticipated to 22 billion devices, due to be connected in the Internet by 2021, the opportunity of being valuable vulnerable to all kinds of of risk will have serious impact on the life of billions of people. Besides that, we need to remember that always we need more and more IP to support the current and the future technology revolution. There we need to work hand in hand to achieve our highest goal, our people desire to be able to harvest the fruit for that progress. We, as operator, decision maker and expert in this world for communication and information have a lot to offer in this field.
Ladies and gentlemen, your meeting today is not just an ordinary occasional meeting. We are looking forward outcome of your discussion and there embark on the future of the Internet and ICT in this part of the world.
I thank you again for granting me this opportunity to address you at this event, wish you all nice and fruitful stay. Thank you.
(Applause)
HANS PETTER HOLEN: So, thank you very much for those words, Your Excellency. We also have a speaker from our local host, Mr. Saleem Albooshi, who we welcome to the stage.
SALEEM ALBOOSHI: Saleem Albooshi, recently appointed as chief infrastructure officer and now my CFO clarified in order to avoid the confusion between the eye between the chief information officer and chief infrastructure officer. We are committed to go for chief officer of infrastructure, COOI. So, welcome to the UAE. Welcome to the city of happiness, to the country actually of happiness, welcome to the country of tourists, welcome to the country of innovation, welcome to the country which embraces technology, welcome to the country which driving in the region actually the leadership of smart initiatives, and as heighted by his excellency, Mr. H.E. Harmad Obaid Al Mansoori, recently the government has appointed a Minister of Artificial Intelligence and I personally will not be surprised at his next announcement of His Highness, our prime Minister, that he will announce a robot in his Cabinet as well. So, I will not be surprised. So thank you again for coming to the UAE. I'd like to thank RIPE for their support for the region. I personally have worked with RIPE since my engagement in the networking 1995, and again appreciate the support of RIPE all the way. This is the second RIPE meeting I remember the first RIPE meeting now I was discussing with Paul, he said ten years. I expected more than 15 years ago, RIPE came to the UAE and again thanks for the effort, thanks for the support for the region from outreach promoting technologies, again, as we have seen as well, the three tutorials running today clearly shows RIPE commitment to the region, RIPE commitment of technology transfer to the region, IPv6 tutorial, IOT tutorial, security tutorial, to enable us for the future. Three key enablers for the future. So, again, thank you very much for the leadership, thank you very much for the strategy and thank you very much for the support for the region. I would like to thank the TRA for their support of promoting telecommunications and the development of telecom in the UAE in general and positioning UAE as a strategy hub for different events and again, this event being in the UAE is an effort from our TRA team.
Fifth, I would like to share with you our commitment to the development of the infrastructure. Over the past more than 11 years we have developed a state‑of‑the‑art infrastructure nationwide. We have more than almost 10,000 km of fibre scattered in the country from border to border. We have invested in submarine cables. We have an infrastructure from access perspective latest 4G and optical geoport infrastructure to the homes, actually the first to implement fibre to the homes in full cities. Again, our core infrastructure, the investment in data centre, with mobile and the development of ‑‑ which is a data centre of hosting its data for technologies, we are again a partner of Smart Dubai, so positioning is not a traditional telecom operators, our strategic communication corporation, EITC, the position to enablers as all innovation happening in the UAE. So, thank you very much again for the support and looking forward to wish you all the best in your coming few days. Thank you.
(Applause)
HANS PETTER HOLEN: Thank you very much for those words, and now I will hand over the stage to Benno, who is Chair of the Programme Committee, and he will talk a bit about the Programme Committee and then take over this session. So welcome.
BENNO OVEREINDER: Thank you, Hans Petter, and local host and our local supporter.
The PC: Hans Petter already introduced us. These are the PC members. And the thing to remember here are the people here, everybody is equal, but some of us are more equal. And these are the people in the blue frame, because these are the people who are elected by you, so, I'll come back to that later.
So, the other thing I want to mention here is, if you see us, give us feedback, so if you see one of these faces around in the corridor, give us some feedback, what do you like? Which do you think could be improved upon? It's very important. And of course our local host representative, Khalid, I welcome him next. I will skip this.
The PC organises the Plenary programme, the tutorials and the BoFs. Only one single thing I want to stress is, over the years we have a very good Plenary actually of high quality presentation, that's because of you, because you submit high‑quality presentations that we can select from, it's about 35, 40% acceptance rate, which is quite good. So keep on submitting good presentations, enjoy the programme for this week, and give us feedback.
Rate the talks, the tools and BoFs, you know how to do that. We will repeat that. You can do that via the website. The thing which is also very important and the last thing I want to mention is the PC elections. Every RIPE meeting there are two seats up for re‑election. Nominate yourself. Talk with us, talk with one of these people here you see on the screen, and if you are interested, if you want to be part of the Programme Committee and do the rating and organise this two days, the Sunday and Monday and Thursday morning, please join us, nominate yourself and to the others, please start voting tomorrow afternoon, I guess.
So, I want to close down this part. And I want to invite Luai to give his presentation on traffic patterns in Saudi Arabia. You are very welcome.
HANS PETTER HOLEN: So, Our Excellency have to leave for another important meeting so I would thank you very much for coming, and thank you so much for helping us with this event. So thank you very much.
(Applause)
LUAI E. HASNAWI: Good afternoon everybody. My name is Luai Hasnawi. I am from Saudi Arabia, from Taibal University. I would like to talk today about the IP traffic in Saudi Arabia. I would like to emphasise first that I'm making this presentation from an academic perspective, most of the numbers you will see, the operators have them, and the communication authority have them but I'm talking from an academic perspective.
So, I'll talk about uncovering IP traffic pattern in Saudi Arabia. I will study latency, routes and hop count.
Me and Dr. Showail, we have performed this study, we are both assistant professors at Taibah University and Prince Mugrin University, Madina.
What's the motivation behind this work? First, we decided to look at the IP traffic in Saudi Arabia and see can the IP traffic handle basic security threats, like for example DOS attacks or cable cuts or these issues, simple cyber attacks, and the second thing is that does the IP network in Saudi Arabia can handle future technologies? Okay, we know that, in future, we can have like IP version 6 claims every sand dust will have an IP, can we handle this enormous number of IP traffic that's coming up.
And the third motivation is about the bandwidth eater applications since 4k video content. And here is our objective for the research. So we decided okay let's first study the pattern, let's see the IP pattern and behaviour in Saudi Arabia and see how the IP traffic is generated. And then we will measure the average latency within Saudi Arabia and across the border of Saudi Arabia. And then we will see the packet loss in IP traffic in Saudi Arabia and lastly, we will provide some recommendations for improving the quality of service and the quality of experience of IP traffic.
Now, we use the RIPE class for this research, actually for this measurements and we mainly use only ping and trace route commands from the RIPE Atlas. At first, we decided, okay, let's see how many probes are there in Saudi Arabia and where are they distributed? And we found out we have only 12 probes in Saudi Arabia and they are distributed in only 4 regional, 4 government regional areas in Saudi Arabia as you can see in the figure.
So, here is our distribution of probes and this is what we are going to do. We are going to study the IP traffic in Saudi Arabia which we call intra‑Saudi IP traffic and then we will have the measurement outside Saudi Arabia which we call the IP traffic outside Saudi Arabia, which we call inter‑Saudi Arabia traffic. And we will study latency and we will trace the IP traffic and then we will measure the packet loss, and lastly, are we ready for IPv6, or let me rephrase that: are they declared to be ready for IPv6? Because we don't know what's going on behind the scene as a researcher.
So, first, we will look at the intra‑Saudi Arabia traffic, we found that we have 12 probes. These probes are distributed between the cities you can see, and this looks at the figure, let's look at this figure as fully connected network but for better visualisation we remove some of the links. Okay.
And here is a measurements that we use for the ping command. So we have 4 packets, each packet has 32 bits, and then the interval is 30 minutes. We use IP version 4 and then there is a start time and end time, date and time.
We found that the latency in Saudi Arabia is quite high. When you see the smallest number of latency that we found is 1.5 millisecond. 175 millisecond we found that is between 2 probes that connected to the same work in the same building, it's for an ISP, I know the name of the ISP, but I'm not going to declare it here. And then we found that we have a very high delay traffic inside Saudi Arabia, which is almost like half a second to send and ICMP packet, which is quite high. And then as you can see, most of the numbers between two different city ranges between 30 milliseconds and 40 milliseconds, except some of the hundreds here and there and there, and so on.
So, then we decide, okay, let's do the other measurement, let's see how the IP traffic is going inside Saudi Arabia. So we use the trace route command to figure out the traffic pattern in Saudi Arabia, and here are the parameters for our study.
We basically use the standard parameters that come with the trace route command. Then we found out some significant results, let's call it some remarks. So, we have one probe that's connected to Sahara Net. When this probe wants to communicate with another probe that's in the same network, the traffic goes to India and France and then back to Saudi Arabia. Okay, so, that's quite unusual, okay, probably we have ‑‑ there is an explanation that I don't have access to it. And then I found out that when I communicate from a probe that's connected to Zain to a probe that's connected to Sahara Net it goes roundabout. That's the first comment that I'm going to comment on the trace route study.
And then, I studied all the probes that's connected, all the probes that's connected to the Internet and I tried to communicate with the probe that's connected to Zain and they found out that all the traffic is directed ‑‑ all the traffic is directed to Kuwait and then came back from Kuwait and I can't find an explanation for that because Zain is a Kuwaiti company, but why would the traffic from Saudi Arabia go outside the country and come back? Okay.
So, that's basically what I found out when I studied a trace route. The rest of the trace route commands are pretty normal, they went inside the countries. But those are the awkward ones or let me say the different ones.
Then we said, okay, let's study the result the same result, the latency and the trace route result in the packet loss and study these results when the traffic is generated from Saudi Arabia to outside Saudi Arabia. So, we decided okay, here is the thing we are not going to use probes any more because those probes, I can't rely on them, sometimes they are on or off. I decided let me go to another academic institution and I got some academic institution from every continent in the world and they came up with these institutions, they are randomly selected. I tried to get well known servers that their firewalls allowed me to use ICMP commands to them. And then I eliminated some of the probes because some of the probes they are connected to the same network from the same building and they all share the same ISP. So, I ended up with eight different probes out of 12 probes.
And here is ‑‑ here is a distribution of the servers that I used and I took this, I made sure that the locations are located in their countries from the RIPE website.
And then the measurements are the same, I use the same measurements, different times, but it's the same. And here is the result. I found out that the delay, the delay to reach some of the universities are really high, like when I tried to reach the university in Australia from Saudi Arabia, the delay is really high, which is not unusual. And the same thing when I tried to reach Ghana but I found out that reaching some of the ‑‑ reaching the UK sometimes, like, not as ‑‑ the delay to each some universities in UK is not as high as reaching another university in Saudi Arabia.
And then I performed the trace route on the same servers from the same probes with these measurements and I found out that I can reach, for instance, the University of Wollongong with a minimum number of ten hops and a maximum number of 25 hops and a similar thing when we tried to reach Pittsburgh, I almost reached Pittsburgh with the same number of HTPS. I found out that reaching South America is kind of high. I am not going to generalise South America, I'm just going to say reaching the University of Chile was quite high.
And then I said to some of my remarks on the trace route command and I found out that reaching Australia is like almost communicating with another ‑‑ with another server inside Saudi Arabia ‑‑ I mean, the number of hops to communicate with the University of Wollongong is slightly higher than communicating with another university inside Saudi Arabia. So that the difference between the number of hops in Saudi Arabia and communicating with Australia is not significant, although the difference in distance is really high comparing 600 km in Saudi Arabia between the probe and the server and 12,000 km between that probe and Australia. So the number of hops is to reach any destination from Saudi Arabia is quite high.
And one more thing that I found, is that all the traffic that goes through different ISNs, but the question is, I didn't find any IXP in the middle, until I see, I communicated with Japan and Australia then I found the IXPs, so I thought okay, here is the thing, maybe if we have some IXPs inside Saudi Arabia or something like that, we have reach those universities with fewer number of hops and as you can see, the traffic, the traffic from the probes, the number of hops that I use to communicate with the university of Ghana was about 20 hops. 7 of these were inside Saudi Arabia. So I need to go to 7 hops inside Saudi Arabia to start going outside the country.
And these are the probes that I was able to ‑‑ that created significant output. So, the minimum number of hops were reached by using this probe, the probe in Medina, and the maximum number of hops were reached by the other probe, which is also in Medina but they all connected to different ISPs. I know the name of the ISPs but I'm not going to mention it here in this study.
What about IPv6? I found out that only 1 probe that declared its readiness for IPv6 and the rest of the probes they are only IPv4 communication, probably they are red for IPv6 but this was not declared in their probes properties.
Then, we came up with these questions. How how is the delay in Saudi Arabia, is it high or normal? Is the delay related to the density of the population? Is there a pattern? What if we had an IXP, would the number of hops perform better or not? And then I run different studies and then I came up with these results.
So, I picked four different countries randomly, UAE, UK, South Korea and Turkey and I run the same ping command to measure the delay and then I found out that the delay is really low in these countries, okay. It's really low and not comparable to the delay in Saudi Arabia, so I found out, okay, here is the thing: the delay in Saudi Arabia is really high based on the studies that I made. And then I said okay, here is the thing: Let's try some of the high densely populated areas in the world and I found out that these are the highest densely populated areas in the world and then I compared the results if with the delay in Saudi Arabia and I found out that although these areas are really really highly populated but the average delay is better to some point than the traffic in my country.
So lastly, I said, okay, here is the thing, what would happen if we had an IXP? Is that going to affect the number of hops? And then I found out that we have two neighbouring countries that have IXPs, one of them is Bahrain and the other one is Emirates and then I run the same command, the same trace route command with the same parameters from probes inside Bahrain and UAE to the University of Wollongong and the University of Tokyo and then I found out that we out perform the results, although they have IXPs and we don't. And lastly I found out that we do have an IXP in Saudi Arabia that was recently launched. I'm not going to talk about it because I don't have that much detail about it. But, as you can see, to reach the University of Wollongong from Saudi Arabia, we needed only 10 hops, but to reach the university of the best number that I can get to reach the University of Wollongong from UAE was 16 hops and 14 hops to reach it from Bahrain. So we out performed them, we out performed the minimum number of hops, but we'll come to the maximum number of hops, they out perform the number of hops than the Saudi traffic.
And I found out, here is the thing: Bahrain, they have an IXP and they have Zain, the same carrier like us. Does the traffic go outside the country, go to Kuwait like the traffic in Saudi Arabia? And then I found out yeah it does go outside the country but it goes to another destination T goes to Singapore. It goes to Singapore to an IXP in Singapore and then came back to Saudi Arabia. It came back to Bahrain. I assume that because they have IXP probably it's going to go to ‑‑ the traffic will be routed inside the country, but it went outside the country to Singapore.
So, here is my major finding. My major finding is the average round‑trip time in Saudi Arabia is really high, half a millisecond was reached in one of the studies, almost half a second which is 500 milliseconds and then the number of hops was really high to reach any destination from Saudi Arabia was really high compared to other numbers. And then the traffic that's routing Saudi Arabia, that's routing outside Saudi Arabia, if there is an explanation that I'm not aware of, then we can declare it, but as an independent academic personnel, I don't have access to this information. IPv6, we are not ‑‑ I'm not going to say we are, based on the information that I have, I see that we are not a hundred percent ready for IPv6. And then when it came to packet loss, I never had an issue with packet loss, they are always zero percent, and by that I conclude my presentation.
Any questions?
(Applause)
BENNO OVEREINDER: Thank you. Any questions?
AUDIENCE SPEAKER: Thank you very much for your presentation and nice to hear details about traffic patterns. I have a question about connectivity inside Saudi Arabia. As far as I know, the country is very long in longitude, so there is a problem in physical aspect of the traffic, so, latency may be long because of it, but anyway, as far as I see there is a problem of poor interconnectivity inside your country, and I hope that commercial interest of people and of business, local business, make other IXPs possible and give birth to next IXP which will make traffic flow faster and make some Internet services possible inside the country. I should say that the commercial interests gives birth to IXP and interconnectivity. This is the way it works. Anyway, I wish the best for your country.
BENNO OVEREINDER: What is your name and affiliation?
SPEAKER: I'm very sorry. I am Filippe Duke from NETASSIST. Sorry for that.
LUAI E. HASNAWI: Here is the thing. These studies are made from an academic point of view. The ISPs, the regulators, they all have good explanation, the data that I got is I measured, I am sure they have data, I couldn't have access to it. Okay. The way they are connected, the connectivity, I'm not aware of. I know the connectivity in different countries because they are well published, but these are the ones that I came up with.
AUDIENCE SPEAKER: Hello. This is Bernd from DE‑CIX which was operated UIX. First, let me thank RIPE for this fantastic SLAs Atlas help. It's helpful. I would like to comment that we have far too less probes in this region, not even in Saudi, all the other countries could have more probes, so please track at the RIPE counter to ask for some probes and install it in the network, this would make life much more easy and coming to a concrete question, would I like to ask you if you have made some special research on the neighbour countries, so how about the connectivity in latency to the neighbour countries like UAE, Oman, Iran, Iraq, so, how much of the traffic is going via Europe? Thank you.
LUAI E. HASNAWI: I made some comparison. I didn't do it with our neighbouring countries. For instance, at first I decided, okay, I want to do it with Egypt, then I find out they only have two active probes. Okay, so I couldn't do it.
BENNO OVEREINDER: I'll close the lines. Short questions, one minute about. Jim.
AUDIENCE SPEAKER: Jim Reid from RTFM. I think what you have really done there is make the business case for a stronger IXP presence in Saudi Arabia and I hope you have discussions with the local stakeholders to encourage them that lots of the traffic that is between Saudi entities is routed in Saudi Arabia and isn't routed via Internet Exchanges in Singapore or Frankfurt or whatever, which I think you might be suggesting is happening at the moment. However, one thing I'm puzzled about is your earlier comments about latency numbers within Saudi Arabia and I wonder if you can correlate that in any way to where the fibre actually runs inside the country? If you are getting very long delays, is it because of the way the fibre is actually organised that that's why the traffic is taking so long to get from one place to another?
LUAI E. HASNAWI: Well, we do have an exchange point, okay, and there will be a presentation, will make a talk about it. Regarding fibres, we do have fibres, but the terrain of Saudi Arabia makes it hard, you know, we have long hoops, in telecom there is long loop and short loop. And it's going to happen one day or another and I'm aware of it, my Ph.D. in fibre optical communication so, it's just too expensive to send people to dig holes and track fibres, and it takes time, it's time‑consuming. But it is happening, they are making fibre in my neighbourhood now.
AUDIENCE SPEAKER: Hi. My name is Nathalie Trenaman, I work for RIPE NCC. Thank you for your very useful presentation. I have more a remark. If you go to that slide that shows where the traffic is going to Singapore, and it's one of the outliers, I did some research myself for a different country and I also had a weird outlier and in my case it was because ‑‑
LUAI E. HASNAWI: The other one was from Bahrain. But that was from Saudi Arabia and it landed on Singapore.
SPEAKER: Anyway, it doesn't really matter. I was just... I had one situation where I had in a similar graph had traffic going somewhere I was not expecting it to go, and the back end, the database used for this is Open IP Plan, and that's an open source geolocation tool and in my case the problem was that somebody put that IP address in a country where it wasn't actually at. So, that basically messed up my data in my research. I'm not saying it's in in yours, but I'm just saying if you use open source geolocation databases, these things can happen. So it's always worth to verify other sources as well.
AUDIENCE SPEAKER: I'm Kaveh, I am in charge of RIPE Atlas, I have two quick comments. One, RIPE Atlas, as have pointed out we have far too few RIPE Atlas probes in Saudi Arabia, 12 or 14 is not a good number. But the way Atlas works is people in the country, or people who have links, me and you and people who are ‑‑ they distribute the probes, so, please talk to me or any of my colleagues, we are more than happy to make an ambassador, if you know people who are willing to ‑‑ this is how it works. So, for any country, especially Middle East, because the coverage of the countries in the Middle East is not as good as we want. Please reach out to us, be more than happy to hand you probes and we hope that you install them and make them active.
That's the first. The second one, actually we have a project, if you search for IXP country Jedi, which we basically pick up ASNs within a country and we ping and trace actually from each probe, from each single AS in the country and we draw a map and geographical map and also basically chart where you can see from each probe to the other probe if the traffic stays within the country, goes outside the country or if it passes through an IXP, and we generate them automatically.
LUAI E. HASNAWI: Is this different than this one? The interactive map?
AUDIENCE SPEAKER: It has a map with lines, it shows the traffic for example Saudi goes to this and this country.
LUAI E. HASNAWI: I have seen it and then I came up with this money, it's easier to present it.
AUDIENCE SPEAKER: I'm not talking about, if you want to compare we do that for all the countries in the world. That's available and it's live.
The final one is actually about the maps, Nathalie is right, that many times, especially infrastructure geolocation is not accurate. So, for end user, it's much more reliable but for infrastructure with the routers, things like that, the IP address geolocation in many cases is not accurate. But RIPE NCC actually has a project followed up on the Mat Working Group, it's called Open IP Mat, based on input from other people and other data sources, we actually are building a database for geolocating Internet infrastructure, routers and hops and things like that. Thank you.
BENNO OVEREINDER: Thank you for your presentation.
(Applause)
JAN ZORZ: Okay. I will take care of the rest of the session, the housekeeping, the usual stuff. Next speaker is Richard Cziva from the University of Glasgow, and he will talk about Ruru, the realtime wide area TCP latency monitoring.
RICHARD CZIVA: Good afternoon everyone. Thank you for the organisers for inviting us. I am also one of the RIPE academic cooperation initiative recipients, to thanks for the programme for funding us to come here. As I have been introduced, I'm from the University of Glasgow and I am going to present to you work that I have been doing with REANNZ, which is a New Zealand education and research network provider.
So, just to introduce our lab, the lab is called Network System Research Laboratory, called Netlab in short. University of Glasgow is our home. It's over 500 years old. Our team is relatively small. We have three academics, four researchers and seven students. We do different project works with BT, Google, and others. Our projects we match computing and some network security, data plane and so on.
Other introduction I want to make is, as I said REANNZ is the operator of the research network in New Zealand and also ‑‑ New Zealand is just there ‑‑ it's even further away than Australia so for the previous speaker I want to suggest to have a look at even the next hop, to go to go to New Zealand, and you will see even higher latencies.
So, this is the network in New Zealand, as you see there are two links that we have to outside world. One was to Sydney and one was to LA. It's important because what I'm going to show you is going to be latency measured on live traffic that goes from Oakland to Los Angeles, so this is just one part of the traffic that goes outside the country.
The company is based in Wellington, which is just there, in the south of the north island, and ‑‑ it connects universities and research labs across the country.
And I have been working with REANNZ before and this is how this project was born. The name. So we named the system Ruru, and Ruru is a bird from New Zealand, it's a native auld and it's called a watch for guardian and our system looks after TCP latencies, so we chose this name for the system.
So, the motivation. So as every network operator is looking after their users by understanding the performance they provide. We identify the problems with current network monitoring tools as they are either being too course grained, meaning that for example they collect poor statistics or they deaggregate data in five‑minute intervals, and in fact the previous measurement was using an active measurement setup where trace route was used to generate a measuring packets. We take another approach where we use a passive monitoring and we try to understand individual user perceived performance, meaning that we designed the system to analyse real traffic without introducing any probe packets. And the reason we did this because there were no easy to use free tools available, or the techniques were slow or constrained you could use for such analysis. Also, as you will see later, we visualise the data, so operators can easily understand if there is any latency issue in their network
So why end‑to‑end latency? One of the reasons is we are seeing an increasing number of realtime applications, for example virtual reality is being introduced for online games, we have financial trust anchors processing that requires very fast round‑trip times between locations, also the architecture requires very low latency, just milliseconds or sub‑millisecond latency for applications such as robotics, tactile Internet and so on. Also, from New Zealand's perspective, it is quite isolated so it does matter to have a look at the latency that users experience as just one routing problem can cause 100 millisecond of delay. For example, if traffic goes to the US and it leaves through Sydney, that will introduce a delay that we are trying to understand and minimise.
So, we are focusing on actual, accurate end to end‑to‑end latency and this is what Ruru does, it's a measurement pipeline that measures this end‑to‑end latency between the source and the destination host. We visualise it in realtime in a world map, and Ruru is entirely open source, the technologies are, as you are probably familiar with these, DPDK, Zero end processing, we have a time series database and we have a WebGL base front end that we developed to visualise all this traffic.
Just to clarify, what do I mean by end‑to‑end latency? So, it's TCP only right now, and in the time it takes to between the first SYN that we see, you can basically take Ruru as a middle box that intercepts all traffic. So, the end‑to‑end latency is the time between the first SYN and then the first ACK from the source. And we are not just measuring end‑to‑end latency, we also save separately and external and internal part of the latency. So if you deploy this at the edge of your network you will have latency results that are, that were inside your network and then you separate the results that were outside of your network.
And, of course, we use some kind of mapping to geographical locations and that's also available for IPv4 only, so, these are the limitations, so it's IPv4 only and TCP only.
In very high level this is how it looks like. For example, if you take two routers, and there is an optical link between them, we used a box like this, which is an optical fibre tap, we tapped the lines and we direct the tapped traffic into a measurement box. We do a per‑flow latency measurement in one part of the system, geographical mapping, the filtering in other part and then we have the front ends.
So, quickly, I would like to just go through the three challenges that we have while designing the system. First of all, how to process the packet. Clearly, if you would just to use tools such as something from the Linux kernel that won't give you the performance that you need as you might have seen this before, this graph just shows the performance from the Linux can or cannot handle, and the blue line is the process 10 gigabit per second traffic and the yellow box shows you that, under 500 bytes, the Linux will struggle to keep up with processing. And we are really focusing on realtime so we wanted to do something more low level than using the a high‑level application.
So there are a lot of things you can do. And this is just some collations of why exactly we need a load leveler, basically kernel bypass. Without going into too much details, these are two different packets sizes calculated of the arrival rate between packets, if you want to keep up with the 10 gigabit per second traffic and we see that for a large packet, you have 835 nanosec between packets, while with 64‑byte packets you have only 67. If you translate it to CPU cycles, you only have 200 cycles to process, 200 cycles to process a packet if it's 64‑byte packets in order to keep up with 10 gigabit per second. And the reason Linux and high level tools will not keep up with this is, for example, if you just take an L3 cache hit, basically with one cache, we use basically almost all of our free cycles.
So as I said, there are many approaches to work on this and we have been talking about this for years. One of them is DPDK that we have used. It's a set of libraries that you can use for fast packet processing, DPDK means data plane development kit. It has a lot of uses for multicore processing, ring buffers, and other drivers, and these libraries you can use to receive and send packets within, with only using 80 cycles or less. So, if you remember 200 cycles, basically with DPDK we can easily keep up with the 10 gigabit per second traffic which we initially targeted.
Just a very high level overview. I'm sure many of you have seen this, the DPDK has that access to the hardware, these cards are usually Intel but there are other providers that support DPDK, the good thing with it is it's isolated in a sense that it won't cause you any sack fold or it won't break the machine. And of course it's proven high performance.
The second challenge we had to solve is how to map all these flows that you are collecting from Ruru to geographical locations because we want to know from where to where the traffic went. And the solution here is not hi‑tec, we used multi‑threaded C programme that uses an offline geo database. That seemed to be quite accurate and they claimed to be 99.5% accuracy.
And we also do ASN look‑ups, and it's not just the geographical location but we also check which AS the IP belongs to. And the reason we do this is because high level libraries such as the libraries that use a rest tool, a distant server will not keep up with thousands of flows per second. So we had to double up this from scratch.
And a search engine is how to visualise all this data because we are dealing with thousands of flow per second, how to put it in a live map which is easy to understand and works.
So obviously we start with these three high level Java script libraries, it didn't work. We had to use some low level and this is why we use WebGL and the Web GL is 3D visitalisation. The libraries, if you are interested, it's Deck GL, which is also used by Uber, Luma GL and React‑Map GL and they are using it to track hundreds of thousands of cars realtime. And as a result we have a visualisation right now that can run 21 frames per second on just on my laptop, and you can probably go even higher, it depends on how many lines we have to process. And I will show you a demo basically we are drawing lines for each individual flow so each line means one flow and the colour of the line means the TCP latency that we measure from end‑to‑end.
So solving these three challenges, this is how Ruru was born. A little bit of details about the architecture. For those who are interested, we also have a paper where we put all the details.
So some of just to mention one thing, which is quite important to put a receiver side scaling to the system to distribute the packets that come in to multiple queues and once we do this, we take processes that run on separate course of the CPU and then consume packets from one queue. And then basically the, from now on, it's multiple threads that we use throughout the pipeline. So, multicore processing is introduced here in the card, which is very important to, for the speed.
The two front ends that we use one is gaffe Anna, open source, it's not us, who developed that, but the one here, the Ruru map, is a custom software that as I said we used WebGL and different libraries.
So, what can we use Ruru for? For example, you can use it to localize faults in wide area N. You can see there is an increase in latency, for example to one of the ASes that you are peering with in networks or one of the ASes that you visit a lot. For example, Amazon or Facebook. You can localise faults in your Internet if you put it on the border of your network, you can even see for example one of your universities or your clients experience a higher latency, that can be an Internet fault in their network because their Wi‑Fi is jammed so basically they are users end‑to‑end latency increases and you can see it as an operator, even if it's not your problem. And of course it can be used for network planning and audited be, just designing the next stage of your network, because you see in live where traffic goes and the latency that the users are getting.
Sop we deployed Ruru just about a year ago and it's going between Aukland and Los Angeles, it's deployed here, and two bugs that I want to show you that we found when Ruru was deployed in production. The one is the 00: 48 bug as I call it, every night at 00:48 there was just a few connections we were experiencing a 4 second lanes. We didn't know what it was. I stayed up all night, in that night it didn't happen but the next night I saw it again and it was just a big spike, the 4 second latency. Then we found out there was a fibre opt gate that started at 00:45 but it took time before it downloaded the new signatures and then it we started itself and just a few connections but they experienced the 4 second latency. So it was good that we could find it with this software. Also we noticed other issues we software switches just by looking at data and the graphs. We did off line analysis, we could find CDNs, basically locations that are providing very low latency. And some characteristics of our user traffic.
Live demo: I am connecting with the VPN to Glasgow but I am accessing the server in New Zealand. That's latency. That's what I call latency. Okay. So this is live right now from New Zealand, so this is part of New Zealand's traffic that leaves New Zealand and goes to the US. And ‑‑ let me just zoom down to New Zealand. This is a 3D map. So we have it in the office also projected. For a more serious application, we can, for instance, if there is an Internet problem see that, for example, Wellington started becoming orange and then red, we can have a look at what's happening with the universities or with our clients in Wellington, and then, for example, we can just eyeball and look at the connections and go to the US. Theories one that is a bit orange. Let's see what's this one? This one was 346 milliseconds to the US, which is about, it could be all right but if you see red lines to the US that basically means 500 milliseconds or more, then we can start looking into the network. Here is a red line. It was 527 milliseconds right now from Waikato to Girard in the US. Of course if it's just a few lines, that's not a real issue, but you know, we have this visual in the operations centre and once you look at it you'll see if there is a serious problem to one of the regions.
And I would like to show you... we can also have a look at any region, so. I want to show you the graph Anna interface, so we are saving all this data into an influx DB, so you can visualise is with graph Anna very easily, and the reason we have this, this is where I found the 00:48 bug that I was talking about. We can filter to a specific country. So, let's see basically the latencies that are coming to the Emirates. It's usually around 300 milliseconds, as you can see, there are a few connection that is resulted in 812. Again we can look at days and months and just see visually, if there were spikes and how many spikes were there. Of course we have an aggregate to look at all of our traffic and see the average latencies. We can see the source ASes that were basically generating all this traffic, so one of the biggest clients is University of Waikato, the traffic is originating from, obviously from New Zealand, and destination AS, is 22% of the time is Amazon, sometimes Amazon servers. I guess this one here is Facebook. This one is what IP location couldn't map to an AS. This one here is Facebook. Destination countries, United States for most of the connections. And we can have this visualisation which I really like. Again, for example, here, we see that there was a 10 second connection, just one, and even in Cuba, there was 4 second connection. As I said, just a few connections doesn't mean much is could be just the user's wireless first hop that's jammed or it doesn't necessarily mean a network problem for REANNZ, but if we see multiple of threes we can go and maybe with a different approach for example trace route and ping or different debugging, we can have a look at the actual problem.
So, more on the paper that I have already mentioned and everything here that I was talking about is open source. REANNZ/Ruru on GitHub, everything is documented there, relatively easily to deploy it. The only requirement is to have a DPDK‑enabled card, which is basically any medium‑ranged server. There's a project page for more information and just to conclude, we believe that user perceived end‑to‑end latency is important. We designed Ruru, which is a full pipeline to do this in a larger scale. We used careful engineering, parallel processing and low level tools to achieve this. So it works for a year now. And it's all Open Source and if you are interested deploying Ruru, let's talk.
And with that, I would like to thank you and open for questions.
(Applause)
AUDIENCE SPEAKER: Hi. Brian Trammell. This is really, really cool stuff. I actually want to grab you at some point and we should go talk off line for a bit. I had one quick question. When you are doing the per flow sampling, are you basically just sampling on the handshake RTT or doing running RTT samples of the flow?
RICHARD CZIVA: We take a hash of each flow, we are not sampling.
AUDIENCE SPEAKER: Okay. But, I mean, you have to measure the sequence ACK number timings so you will get multiple measurements per flow or are you just getting a single handshake?
RICHARD CZIVA: A single handshake, the first one. That's something I'd like to improve.
AUDIENCE SPEAKER: Let's talk off line.
AUDIENCE SPEAKER: Kaveh: I just want to say great work, I was thinking of putting one of these in front of our servers, to see the traffic life and maybe even show that to the Board. We need to you can at that. Great work. Thank you.
JAN ZORZ: Are there any other questions? Okay.
AUDIENCE SPEAKER: Alexander Azimov, Qrator Labs. I do have a very ‑‑ first of all, thank you very much for this work, it looks really great. Thank you. And just correct me, all your is developed with the GitHub including the look‑up library?
RICHARD CZIVA: That's free so I haven't uploaded the database itself but from IP to location you can get it for free and just put the binary file that they give.
AUDIENCE SPEAKER: But the multi‑threaded ‑‑ is in the ‑‑
RICHARD CZIVA: Yes, yes.
JAN ZORZ: Anyone else?
Thank you very much.
(Applause)
All right, we have a couple of administrative things to say. So, please, Benno mentioned that there are elections for the Programme Committee. If you want to nominate yourself or somebody else, please send an e‑mail to pc [at] ripe [dot] net. Send a bio and the motivation why you want to be on the RIPE Programme Committee and then we will have elections.
Second thing: Please submit lightning talks. If anything comes up in your mind, please just submit the lightning talk, we have the submission system and we will rate it and everyday announce the new lightning talks.
Then, at 6 p.m., we have this diversity BoF, that will be in the side room, but we will also have sort of like the unofficial BoF that will focus on measuring the health of routing ecosystem and their players. This is the unofficial BoF because we did not publish any agenda and here things are a little bit different. We will also have not the pick‑up Task Force this time because the chairs were late with the agenda and we just had one pick‑up published, that's RIPE 690, so we are postponing the pickup Task Force to the next RIPE meeting, but instead we will have the measuring the health of the routing ecosystem. Whoever is in interested in this BoF, we don't know where in this hotel it will be, it will probably be with a bar, so who is interested in discussing this, please find Benno or me and we will tell you afterwards where the place will be.
Now we have a coffee break, and please be back in less than half an hour. There will be a bell going around. Thank you very much.
(Applause)
(Coffee break)
