The Men & Mice Blog

Men & Mice at Cisco Live 2019: Wired for Change

Posted by Men & Mice on 6/12/19 10:17 AM

We live in a software-defined world. Whether we talk about multicloud or DNS privacy, bits and bytes are sorted, sent, and protected using software.

Today’s enterprise and large scale organizations are looking for software overlay solutions that can maximize the value of their infrastructure investments while positioning for future innovation. Many of them also rely on Cisco.

New best practices

clus-2019-cisco-dhcp-menandmice

In one of the best attended ThinkTank sessions of the day, Men & Mice North American Director of Sales Paul Terrill explored new best practices at Cisco Live in San Diego.

Paul's talk focused on how to adapt hybrid network strategies to take advantage of service-native features in all IP infrastructure solutions, whether on-premise, cloud or multicloud. The common pain points in adapting hybrid and multicloud network strategies resonated well with the audience: the potential loss of access control assignments, lost time and staff resources during migration processes, and compatibility hurdles between multiple services are all challenges today's network professionals encounter often.

This being Cisco Live, Paul explored the advantage of Cisco IOS DHCP against other solutions, as well as where most hybrid and multicloud migration strategies go off the rails. He finished with discussing the API-shyness of IT decision makers (and why they should embrace them instead) and why homegrown solutions are no longer acceptable.

We will share links to Paul's presentation and the session recordings as soon as they become available.

Everything has changed

Yet nothing’s different. At the end of the day, software-defined or not, data is still sent and received by computers and still goes through wires, switches, and routers. On-premise solutions still matter.

But compared to today, networking used to be simple. With the Internet of Things and Edge, networks fuel and permeate everything in our world (and soon, out of this world). 

To accommodate such explosive growth, innovations like cloud computing have grown in prominence at an exponential adoption rate. And with cloud technology maturing to meet the strict regulations of enterprise-level businesses, the way we think about networking has shifted.

In our journey to simplify and secure increasingly complex networks, we also have to be aware of the need for compatibility between on-premise and cloud services, and how that impacts our future network architecture choices.

Future-ready IP infrastructure solutions

Men & Mice continues to be a leader in DNS, DHCP, and IP address management, as we've been for nearly three decades. We’ve worked with an industry-horizontal array of customers and gained deep insights into networking best practices as a result. We also recognize the widespread presence and critical importance of Cisco hardware in enterprise networks.

With products like Umbrella, Cisco is continuing to bring network infrastructure innovation to larger audiences. By utilizing the Men & Mice Suite with Umbrella, Cisco customers gain the advantage of being able to control internal DNS resolvers, numbering anywhere between dozens to hundreds, in one fell swoop. In addition, proper visibility quickly highlights servers not properly configured.

To learn more about the Men & Mice Suite, contact us or download your free trial below.

Men & Mice Suite Free Trial

Topics: Cisco Live, Cisco IOS, Paul Terrill

World IPv6 Day 2019 (plus a podcast!)

Posted by Men & Mice on 6/6/19 9:50 AM

June 6th, 2012 (or “6/6”) saw the World IPv6 Launch Day. Today we celebrate the 7th anniversary.

For those in need of a quick cheat sheet, here’s ours.

(Mind you, this is not ONLY a cheat sheet, but also doubles up as a lens cleaning cloth. Come by our booth at  Cisco Live in San Diego to pick one up.)

Beyond that quick reference, what’s all the fuss about this old-new networking technology? What has changed since it’s been around (from the 1990s)? What hasn’t? And where do (or should) we go from where we are now?

To IPv6 or not to IPv6?

That is the question. For what it’s worth we, and literally everybody we spoke to at RIPE 78, are for IPv6.

That said, there is legitimate criticism against it. More often than not, however, it tends to be rooted in shortcomings of implementation, misunderstandings in adoption strategies, or just general reluctance toward the work involved in the switch.

Large tech companies have adopted IPv6 whole-heartedly. ISPs, cloud providers, and data centers have been offering IPv6 for a while. Microsoft has been at work getting rid of IPv4 addresses in their offices for years. Google even keeps a public chart of IPv6 adoption amongst its users:

Screenshot 2019-06-06 at 10.28.10

Bottom line is: adoption is on the up, but it’s still spotty at best. And it is true: IPv6 isn’t perfect. But then again IPv4 isn’t, either. It will not get any better, though, if we don’t dedicate effort to perfecting it through practice.

Fun fact: IPv6 addresses are free. IPv4 addresses go for $20+ a piece and that price keeps rising.

It’s evident that, various inventions and initiatives notwithstanding, we’ll likely soon be out of IPv4 addresses. Never before have there been so many connected devices, from smartphones to cars, from smart thermostats to smart toasters. IPv6 is an inevitability.

What can we do?

Introducing ‘resolv.pod’: a DNS podcast

We can, and most definitely should, discuss and evaluate our options regarding anything and everything affecting the future of the networks we depend on. Attend conferences, read papers, draft strategies.

To that end, we’re happy to announce that we are launching a podcast aimed at sharing with you the mindshare we have access to.

resolv.podOf course, as is clear from the name of our podcast, the focus won’t be on IPv6 exclusively, but rather anything and everything related to DNS, DHCP, and IPAM. Facilitating discussions about IPv6, amongst other things, and giving listeners fuller context from experts in the field, are the DNAME of the game (OK, name - just couldn’t resist).

As luck would have it, we were fortunate enough to grab a conversation with Geoff Huston, Chief Scientist at APNIC (Asia Pacific Network Information Centre)  in the lovely Reykjavik sunshine at RIPE78.

So to celebrate World IPv6 Day, why don’t you sit back and listen to our very first episode featuring Geoff talking networking highs and lows with Men & Mice’s Carsten Strotmann? It’s sure to entertain - and inform - in equal measure. Happy  World IPv6 Day!

Find resolv.pod on your favorite podcast platform:

More interviews and discussions coming up in the next weeks! Let us know what you’d like to learn more about via the podcast email, our social media channels, or as a comment below.




Topics: IPv6, DNS, podcast, resolv.pod, IPv6 Day

Men & Mice @ Cisco Live 2019: New Best Practices for Future-ready Hybrid and Multicloud Networks

Posted by Men & Mice on 6/5/19 11:28 AM

 

Cisco Live San Diego: we’re coming! Find us at booth 2234 for all your DNS, DHCP, and IPAM needs, plus sweet swag from Iceland!

Copy of Copy of Booth #2432Whether you’re attending Cisco Live or not, chances are your enterprise or large organization is well into developing or implementing its cloud strategy. Further, you’re likely capitalizing on a number of cloud services across multiple platforms.

This year at Cisco Live, we’ll have Paul Terrill, our North American Director of Sales Operations, taking the Think Tank stage for a look into what best practices you can adopt today to get your environment ready for the hybrid and multicloud networks of tomorrow.

With more than a decade of experience delivering software solutions that meet the diverse IP infrastructure needs of some of the world’s largest multinational enterprises and government organizations, Paul is an expert in identifying, and solving, large scale network management challenges.

Here’s a sneak peek at Paul’s talk.

Adopting best practices for a future-ready network

Scheduled for Monday, June 10, 03:30 PM - 04:00 PM, PDT, at SDCC - World of Solutions, Think Tank 2,  Paul’s talk will focus on the challenges organizations face in a cloud-native world, the solutions that transform networks into a future-ready state, and the pitfalls to avoid along the way.

During the session, Paul will explore new best practices and the advantages to adapting hybrid network strategies to take advantage of service-native features in all IP infrastructure solutions, whether on-premise, cloud or multicloud. Specific attention will be given to some of the common pain points in adapting hybrid and multicloud network strategies, such as the potential loss of access control assignments, lost time and staff resources during migration processes and compatibility hurdles between multiple services (and how to overcome them).

Additionally, Paul will describe in detail the advantage of Cisco IOS DHCP against other solutions, as well as where most hybrid and multicloud migration strategies go off the rails. He’ll also be speaking about why IT decision makers need not fear APIs (in fact, why they should embrace them) and why homegrown solutions are no longer acceptable.

Made with your infrastructure in mind

We understand the importance of visibility, control, automation, and security — and also how challenging those can be in complex, hybrid IP infrastructures. Men & Mice provides API-driven DNS, DHCP, and IPAM software solutions to global enterprise, education, and government organizations.

Men & Mice also recognizes the widespread presence and critical importance of Cisco hardware in enterprise networks. With products like Umbrella, Cisco is continuing to bring network infrastructure innovation to larger audiences. By utilizing the Men & Mice Suite with Umbrella, Cisco customers gain the advantage of being able to control internal DNS resolvers, numbering anywhere between dozens to hundreds, in one fell swoop. In addition, proper visibility quickly highlights servers not properly configured.

Questions?

While in San Diego next week, come and listen to Paul’s talk, and/or visit us at booth 2234 throughout the event. You’re welcome to fire away with whatever questions come to mind - our experts will be on hand to help you solve your unique enterprise networking pain points.

Topics: DNS, Cisco Live, hybrid network, Cisco IOS, multicloud

The ABCs of DNS: a select glossary from the Men & Mice training archives - Part 2

Posted by Men & Mice on 5/31/19 7:46 AM

dns a-z 2-1Continuing our glossary of DNS tips & tricks, we’re covering the letters D, E, and F this time.

DNS ALERT

Our popular DNS & BIND Week, DNS Fundamentals and DNS Advanced courses are all registered to run June 20th to June 24th, in Reston, Virginia, USA. Still want to join in? All info on our training page

D is for “dig”

Dig is the Swiss army knife of network tools. It's got so much functionality, it’d be next to impossible to cover it all, but here’s a taste:

  • find your IP address using: dig @ns3.google.com +short o-o.myaddr.l.google.com txt
  • relatedly, you can make an alias in your .bashrc file: alias myip='dig o-o.myaddr.l.google.com -t txt +short @ns3.google.com'
  • you can use dig +trace <domain-name> to follow all delegation from the root down.

And if dig isn't available, you can use one with a web interface (sometimes called a DNS Looking Glass), such as https://dns.bortzmeyer.org/[URL]/[TYPE] (for example https://dns.bortzmeyer.org/menandmice.com/AAAA).

Remember, friends don’t let friends use nslookup.

E is for “error-free config files”

To err is to be human. Sometimes a typo sneaks into your configuration files. (Unless you’re using Men & Mice, in which case validation is automatic.)

A quick way to make sure everything’s in order is to run named-checkconf -z to test all zones inside the named.conf file. (Note that the command checks the validity of the master zones, and not the configuration file itself. To check the file itself use named-checkconf <path to named.conf>.)

F is for “FQDN”

FQDN stands for ‘Fully Qualified Domain Name’ and you need it for a number of things. It’s the human-readable address that the DNS resolver translates into its corresponding IP address.

The FQDN is made up of three or more parts (called labels):

  • root (the trailing dot at the end)
  • TLD (such as .com, .net, etc.)
  • domain (such as menandmice)
  • host (such as www, info, etc.)

Each label is a string between 1 and 63 characters (letters, numbers, and dashes), and the total length of the FQDN is capped at 255 characters.

To find the FQDN of your machine:

  • on Windows: Start > Programs > Administrative Tools > Active Directory Domains and Trusts (or echo %COMPUTERNAME%.%USERDNSDOMAIN% in the command line)
  • on Linux & MacOS: hostname -f (on Linux you can also use hostname --fqdn)

Want to learn more?

This series is bite-sized (almost fitting a DNS query) — but it’s just the tip of the iceberg. A lot more is said (and done) in our DNS training program:

  • If you’re new to DNS, we offer the DNS & BIND Fundamentals (DNSB-F) course. It’s part of the DNS & BIND Week (DNSB-W) and serves as a shorter introduction to the world of DNS and BIND.
  • If you’re already familiar with the basics, the full five-day DNS & BIND Week (DNSB-W) course takes you deeper into DNS, including a heavy emphasis on security, stopping just short of DNSSEC (for which we offer a separate course).
  • And if you're looking for even more, we offer the DNS & BIND Advanced (DNSB-A) program, getting into the deep end of things.

Check out our training calendar for 2019, and reach out to us with any questions.

Topics: DNS, networking best practices, dig

The RIPE-javik logs: Day 5

Posted by Carsten Strotmann on 5/26/19 11:06 AM

ripe day 5carsten@menandmice:~$ cat ~/ripe/ripejavik-day5.txt | blog-publish

As RIPE 78 came to a close, it was time to reflect and to forge plans for the future.

The last day of RIPE 78

In the final plenary session of RIPE 78, Theódór Gíslason from Icelandic security company Syndis, talked about current threats on the Internet and that many users underestimate the security issues. He underpinned this statement with some examples of how attackers can find detailed information on the victim through public information like commits on Github or Facebook, and that data breaches are getting more and bigger.

One could say that most of the information in the presentation wasn't new to the RIPE audience, and Theódór Gíslason was somewhat surprised when he asked the audience who is using Facebook and only a few hands went up. The RIPE audience is a special case.

Later on, it was Roland van Rijswijk-Deij’s turn again to take the stage. Today he was reporting on historical data on RPKI, the Resource Public Key Infrastructure securing the Internet’s routing system. The RIPE NCC has archived historical RPKI repositories and Roland used the "Routinator" tool to analyze how RPKI usage has changed over time. For example, he found that the average prefix size in RPKI is decreasing over time for both IPv4 and IPv6.

Richard Nelson from the Faucet Foundation presented on the open source OpenFlow controller with the name "Faucet". Faucet is targeted at enterprises that want to move router and switch management away from closed network equipment vendors into OpenFlow Hardware/Software. Richard reported on their real world implementation of the Faucet system at the Super-Computer Conference 2018 in Dallas, TX.

Before RIPE Chair Hans Petter Holen officially closed RIPE 78, there was a challenging online quiz titled  "Are you up to the Level of RIPE 78?" which was organized by Fernando Garcia. RIPE meetings are often exhausting, quite challenging, but also lots of fun!

A final note

RIPE 78 was the second largest RIPE meeting ever, and for me personally it was one of the best RIPE meetings I've attended. It had great presentations, a good location (Hotel Nordica) and food and very nice weather in Reykjavik. I have been told this has been one of the warmest weeks in May for years. Must’ve been the hot topics at RIPE 78.

And then there was the "Group of Secrets" (aka Secret Working Group), but the report from that group is a secret and I'm not allowed to tell you anything about it. If you want to know what is going on in that Working Group, you will have to come in person to RIPE 79 in October in Rotterdam, NL. See you there!

A note from the editors: RIPE-javik may be over, but not done

Thus concludes our RIPE 78 coverage, but not our investigation of issues raised or following up on conversation started.

In the coming weeks and months, we’ll be returning to these topics frequently. We’ll deep-dive into issues on the blog, and we’re also preparing a podcast series, starting with interviews (conducted by Carsten) with prominent speakers and attendees at RIPE.

We’ve learned a ton this past week. But we’re also interested to hear your feedback: what did you find the most interesting? What new development are you the most excited for? We’re listening!

Topics: DNS, Open Source, Security, network security

The RIPE-javik logs: Day 4 - Part 2

Posted by Carsten Strotmann on 5/25/19 2:25 PM

ripe day 4_2carsten@menandmice:~$ cat ~/ripe/ripejavik-day4.txt | blog-publish

Because it was so filled with information, our coverage of Day 4 of RIPE78 has been divided into two parts. Read Part 1 here.  

DNS Working Group

During the DNS Working Group session, I gave two talks: first one on an Overview of the DNS Privacy Software Landscape and then another “lightning” talk on unwind, a validating DNS recursive nameserver.

At the beginning of May 2019, I started a survey of software projects implementing the new DNS privacy protocols: DNS-over-TLS (DoT) and DNS-over-HTTPS (doH). My questions were:

  • Are there software projects out there that use DoT and/or DoH?
  • In which year did development start?
  • Which programming languages have been used to implement the software?

The results of the survey were presented to the RIPE DNS Working Group and are as follows:

  • Of the 46 projects I found on Gitlab and Github, 21 were implementing DoT, 32 support DoH and 7 projects support both protocols.
  • The majority (29) of the projects implemented the new protocols in 2018 and there are 9 new projects this year. At the moment, I am finding 1-2 new projects every month.
  • A large number of newly created projects use the Go programming language (17), while most established projects that added DoT/DoH to their products use C/C++ (15). There were also projects using Rust, Python, Ruby, Java and JavaScript (via NodeJS).

I was also interested in the liveliness of the projects and looked if there was any activity in the project over the last half a year, e.g. new code checking or issues tracker activity. The majority of projects are active (32) while some are dormant (14). The complete list of projects can be found here.

In the second talk, I provided some information on "unwind", a DNSSEC validating resolver for laptops running OpenBSD. For mobile computers, it is a challenge to get a secure DNS name resolution, as most DNS resolvers in wireless networks don't do DNSSEC validation and are not trustworthy. Unwind implements a DNS resolver that runs on the local machine, listens to the loopback IP addresses and either does direct DNS resolution into the Internet, or forwards the request to a trusted resolver via DoT or classic DNS (UDP/TCP).

Many WiFi networks have a captive portal that prevents direct access to the Internet, and therefore to the DNS of the Internet. unwind can be configured to detect such a situation and will switch to the DHCP supplied resolvers, with the sole purpose of getting through the portal. Once the direct access to the Internet is available, unwind switches back to secure DNS communication.

DNSSEC keytags

In the next presentation, Roland van Rijswijk-Deij (NLNetLabs) presented his research on the DNSSEC keytags he found in the OpenINTEL dataset.

Every DNSSEC key has a 16-bit number (between 0 and 65525) that helps DNS resolvers to find the correct key in a DNSSEC signed zonefile. The keytags are generated by applying a simple and fast algorithm (first standardized in RFC 2535).

In 2016, Roy Arends from ICANN already noted that the keytag numbers are not evenly distributed. This is because RSA DNSSEC keys have a structure, and some parts of the key are not random. Roland took the large data collection he has in OpenINTEL and looked at the RSA DNSSEC keys seen there.

The real world data confirms the observations in the initial experiments with DNSSEC keytags: for RSA DNSSEC keys, some keytags are never generated, and the numbers follow a structure. Also, the crypto-library used to generate the RSA keys influence the key tags, as OpenSSL has some safeguards for weak keys that other libraries don't implement.

Next, Roland looked if there are keytag collisions, where the same keytag appears in a zone twice or more for different keys. He found very few collisions, actually less than predicted by theoretical probability.

Then he tested what would happen if keytags are generated by a different algorithm that guarantees uniform distribution of the numbers (in this case CRC16). Turns out then there would be more collisions (still very few, but more nonetheless). Possibly, and without aiming to, the authors of the DNSSEC RFC chose a better algorithm.

RIPE-javik winding down

This concludes the second to last of our daily reports on RIPE78, but by no means does it signal the end of our RIPE-related coverage. Stay tuned for Day 5 tomorrow, and much more deep industry talk in the weeks to come.

Topics: DNSSEC, DNS, DNS privacy, DNS-over-HTTPS, unwind

The RIPE-javik logs: Day 4 - Part 1

Posted by Carsten Strotmann on 5/24/19 12:03 PM

ripe day 4carsten@menandmice:~$ cat ~/ripe/ripejavik-day4.txt | blog-publish

Day 4 of RIPE78 was so jam-packed, we had to split it in two. Here’s Part 1!

IPv6 Working Group

Day 4 of RIPE78 started with the IPv6 working group. Geoff Huston discussed his measurement engine that focuses on using "ads" delivered to browsers in order to look into the reliability of IPv6 connections. While the IPv6 failure rate has gone down since early 2017 with 4%, it is now at 1.4%. Somewhat better, but still pretty bad.

Geoff found out that mobile networks deploying 464XLAT usually have more stable and reliable IPv6, than others using NAT64/DNS64 or other stateful IPv4-to-IPv6 translation mechanisms. IPv6 reliability appears to be exceptionally bad in Vietnam with a 6-10% failure rate.

Because of the Happy Eyeballs implementations in browsers, end users possibly don't notice the breakage except for a slight delay in establishing the connection. This is both good and bad: while the users are shielded from experiencing the issues in their ISP’s networks, the provider is also not incentivized to fix the issues. (As “it works”). Other countries with non-optimal IPv6 networks are Panama, Venezuela, Morocco, Bangladesh, and Turkey. Even China, with its experience in IPv6 networking, has a higher than average failure rate.

Another artifact Geoff found during his research is the fact that some networks route their IPv6 traffic differently (and often worse) than the IPv4 traffic. At some point between November 2016 and December 2016, all IPv6 traffic from and to India was routed via networks in Great Britain.

In the next talk, Enno Rey and Christopher Werny from ENRW shared their experience with IPv6 on WiFi hotspots. They are working on a project to provide IPv6 on up to 3,000 wifi hotspots in supermarkets and shopping malls all across Europe. After their evaluation of IPv6 support in common applications used by customers of these hotspots, they decided to deploy IPv6-only with NAT64/DNS64. (WLAN 100% IPv6, IPv6 to IPv4 translation at the gateway to the Internet.)

MulticastDNS and other multicast IPv6 communication are a problem for wireless networks, as the access point needs to distribute the multicast message to all clients in range, and needs to use the oldest WiFi protocol to be able to reach legacy clients. Using the older wifi protocols blocks the air for other traffic for a longer time. Enno and Christopher recommend tuning and throttling multicast traffic in IPv6 enabled WiFi networks to minimize this effect. The IETF is aware of the issues and is working on adjustments to the IPv6 protocol family to make IPv6 more WiFi-friendly.

IoT

Over in the IoT working-group, Jan Zorz reported on his attempt to build a "smart home" and gave insight into his design choices. Being an engineer, and also because of privacy concerns, he does not want to use "off the shelf" smart home devices that send sensitive data into the cloud and whose functionality is dictated by the vendor.

So Jan started to build his own smart devices. As the central management hub, he first started his experiments with a Raspberry Pi, but will switch to a more powerful 64bit x86 desktop mini-PC for production. Jan reported that he did not initially really know what he wanted and would expect from a "smart home" system: that insight developed over time while experimenting with different smart devices.

He encourages everyone to do some experimentation before deciding on a particular smart home technology. If you want to hear about the differences between Z-Wave vs. Zigbee, or which home automation software might be the best, have a look at the video recording of his talk.

Next up on the stage was Jelte Janssen from SIDN Labs talking about the SPIN ("Security and Privacy for In-home Networks”) project. The project is developing software tools that help end users to get insight into the network communication of IoT devices in the home and enable the user to better protect the home network. One tool from the SPIN project is the traffic monitor that shows DNS queries and data traffic in the local network, showing graphically to whom the IoT devices talk.

The goal is to get the SPIN tools in the default install of CPE (customer premise equipment, such as home routers) devices. In the same talk, Peter Steinhäuser from Swiss CPE Firmware developer Embedd reported on his company’s work on integrating SPIN in OpenWRT (a popular open source home router firmware based on Linux).

SIDN Labs has installation instructions for people who would like to test-drive SPIN on their existing OpenWRT based routers. Please note that SPIN is still in development and has some rough edges. However, the project would be happy to get feedback (and pull requests) from actual users.

Part 2 coming soon

Watch this space for the second part of our day 4 coverage on RIPE 78.

Topics: IPv6, IPv4, DNS, IoT

The RIPE-javik logs: Day 3

Posted by Carsten Strotmann on 5/23/19 7:11 AM

ripe day 3carsten@menandmice:~$ cat ~/ripe/ripejavik-day3.txt | blog-publish

Wednesday was a hands-on kind of day at RIPE 78. Attending the OpenSource Working Group yielded lots of interesting information, and we’ve interviewed some RIPE 78 participants for our upcoming podcasts. (Watch this space!)

Open Source Working Group

The Working Group started with two different solutions for a similar task, both very interesting.

The first presentation was about building Network Labs using OpenSource tools. Wolfgang Tremmel from German Internet Exchange DE-CIX reported his experiences with using Docker Linux containers to build a training lab for BGP training. He used a Docker container with FRRouting (an open source routing software rooted on Quagga) and exposed the terminal command line of each container via ttyd to the net.

In this configuration, the training participants only need a web browser to access the lab machines. The lab can either run local in the training room or on some cloud service. Getting IPv6 to work with Docker can be challenging, and Wolfgang ran into problems there. I personally would recommend podman or systemd-nspawn as an IPv6 friendly alternative to Docker.

In the same presentation slot, Sander Steffann talked about his experiences with his router labs. While the focus in Wolfgang’s training is the routing protocol itself (and less the routing software used), Sander has a lab that allows the students to try out real commercial router software such as Cisco, Juniper, or Microtik.

Sander is using the GNS3 project that is able to emulate or virtualize commercial router hardware to run the router firmware unmodified. While GNS3 itself is open source, the router firmware needed is not. Emulation is costly, especially for more modern router machines, so his lab needed very powerful machines. Sander combined GNS3 with a nice, web-based management system that would display instructions and information about the routing labs.


The second presentation was from Max Rottenkolber, who was talking about his open source project, a high-performance VPN solution for x86_64 machines. This Site-to-Site VPN software is called Vita and is built upon Snabb, a high-performance network stack running in userspace.

While it is running on top of Linux, it does not use the Linux network stack, instead accessing the network cards hardware from userspace directly. While doing this, Snabb can be used to create applications that are very optimized for network throughput. Vita (and Snabb) are mainly built with the Lua programming language, and the code is compiled to optimized x84_64 machine code using a Just-in-Time (JIT) compiler. Because Vita is bypassing the kernel, it can fully control the hardware and squeeze maximum performance out of the system.

The project is still in development, and the medium-term goal is to be able to encrypt 100 Gbps line-rate traffic (with 60byte packets). Because VPN gateways running Vita are dedicated servers, and because all networking is done in userspace, almost no kernel syscalls are used and the system's performance is not affected by the mitigations for the Intel CPU problems such as Spectre, Meltdown, and others.

Lightning talks

In the lightning talks session, Sander Steffann was asking the RIPE community for help with the NAT64check website he operates. The service allows users to enter the URL of a particular website, and run tests over IPv4, IPv6, and NAT64 in order to check:

  • whether the website is actually reachable in each case,
  • whether identical web pages are returned,
  • and whether all the resources such as images, stylesheets, and scripts load correctly.

Sander is looking for people who are interested in joining the team that keeps this service running.


Next, Maria Jan Matejka from CZ.NIC presented an update on new developments around the BIRDv2 open source routing daemon. BIRD is a dynamic routing daemon running on Linux, BSD and other systems and implements many routing protocols like BGP, OSPF, Babel and more.

The new version has custom route attributes, a filter benchmark tool and will become faster filter in the future. There was also a "dirty hack" presented on how to auto-reload a route as an RPKI change.


The working-group closed with a discussion on industry hackathons, with presentations on both experiences from the IETF hackathons and the RIPE hackathons.

More coverage (And a podcast!)

RIPE 78 is now in full swing, with conference events and lots of off-site discussions, sight-seeing, and social happenings. We’ll continue our daily briefings throughout the week, but we’re also working on a more in-depth project: a podcast digging deeper into all things DNS, DHCP, and IPAM.

Make sure you follow Men & Mice’s social media channels and blog for the announcement!

Topics: Open Source, RIPE 78, VPN, workshop, routing

The RIPE-javik logs: Day 2

Posted by Carsten Strotmann on 5/22/19 8:31 AM

ripe day 2

carsten@menandmice:~$ cat ~/ripe/ripejavik-day2.txt | blog-publish

The second day of RIPE 78 started with the plenary, and three presentations on the topic of Distributed Denial of Service (DDoS).

DDoS

DDoS attacks are an increasing risk on the Internet. Mattijs Jonker from the University of Twente explained how DDoS attacks work. His research has revealed that many businesses have all their Internet services (website, mailserver, etc.) in a single network. In case of a DDoS attack, all services are impacted. He counted 31 thousand websites, 3.5 thousand mailservers, and 323 DNS servers that are on a single network and would suffer in case of an attack. An alternative IP address from a different network (autonomous system/AS) would make the services more resilient.

Matthias Wichtlhuber from the German Internet Exchange DE-CIX found that DDoS attackers only use certain protocols for their amplification attacks:

  • unspecified (Port 0)
  • NTP (Port 123)
  • LDAP (Port 389)
  • DNS (Port 53)
  • Chargen (Port 19)
  • Memcache (Port 11211)

Filtering these ports (in transport networks) will stop most DDoS attacks. The problem is that most ISPs cannot do fine-grained filtering. Most can only filter on networks or IP addresses, which blocks all traffic from or to a certain machine. DE-CIX has developed a new fine-grained black-holing system for DDoS attacks that is currently in beta testing.

Koen van Hove, also from the University of Twente, presented the DDOS clearinghouse: a project to collect data of DDoS attacks in a central place. The aim is to be able to research DDoS attacks and develop fast responses to them. The DDoS clearinghouse collects network measurements, identifies DDoS attacks across networks with unique fingerprints, and stores this data in a database (DDoSDB). From the database, attack information and metadata can be retrieved to help users feed fingerprint signatures into their network systems to stop DDoS attacks.

DNS

After the morning break, the main topic was DNS. David Huberman from ICANN discussed the root server system. After talking about the history of the DNS root server system, he explained that there has been no process so far for selecting new root server operators.

With over 1.120 root server instances in the world, 340 of which are in the RIPE region, the root server system is stable and there is currently no need to add additional root server operators beyond the 12 that run the 13 logical root-server addresses. ICANN is not working on a defined governance model for the root server system.

OpenINTEL

Next on the stage was Roland van Rijswijk (NLnet Labs) presenting the OpenINTEL project he has contributed to. OpenINTEL is a massive active measurement system that sends 218 million DNS queries per day from several vantage points on the Internet, resolving a defined set of DNS names. The results will be collected in a big database (Big Data helps to get research funds these days), which so far contains 3.1 trillion results since the start of the project in 2015.

The OpenINTEL system allows researchers to search for various kinds of interesting data: parent-child TTL mismatches, distribution of authoritative DNS-Servers in different AS networks, or even silly things stored in DNS TXT records (like funny IPv6 addresses or private cryptographic keys). The project can be found at https://openintel.nl

KSK roll

Like always, Geoff Huston (APNIC) delivered a highly entertaining talk, this time about the KSK roll in October 2018.

Officially, there was no impact seen for the DNSSEC validating resolvers. But some operators, like EIR in Ireland, have missed all notices about the roll in the two years leading up to it and failed to change the trust anchor of their DNS resolvers which lead to a full-day outage of their DNS resolver services. Other smaller operators were affected as well, some of which fixed the issue by disabling DNSSEC. All except two have re-enabled DNSSEC after fixing their DNS resolver configurations. Geoff also noted that the DNSSEC trust state signaling (RFC 6975 and RFC 8145) does not work reliably to detect broken KSK rolls in the root zone.

Migration from IPv4 to IPv6

In "Get Ready for Mixed World: Economic Factors Affecting IPv6 Deployment", Brenden Kürbis and Milton Mueller from the Georgia Institute of Technology talked about the economics behind network migration from IPv4 to IPv6.

The problem of IPv6 is that it is not possible to switch off IPv4 right away. Instead, IPv4 must be kept enabled for some amount of time (dual stack deployment). The cost generated due to IPv4 depletion will stay, the cost of introducing IPv6 will come on top. Only after some years will the cost benefits be visible. Depending on the growth pattern of the company and the networks, the first cost savings can appear as early as 4 to 10 years. Larger companies will have more benefit from IPv6, while smaller companies will not see economic benefits. In the following Q&A session, people from the audience challenged some of the assumptions in the research that generated this report.

DNS flag day

In the last DNS talk of the day, Petr Špaček from CZ.NIC and Ondřej Surý from ISC gave some insight into the DNS flag day in February 2019.

DNS vendors (Bind 9, Knot, PowerDNS, Unbound, and others) and large DNS resolver operators (Google, Cloudflare, Quad9, etc.) disabled workarounds for broken EDNS implementations. The workarounds were developed to help with DNS servers on the internet that had faulty implementations of the DNS protocol. However, because the workarounds existed, the operators of these faulty servers had no motivation to fix their systems. The cost of developing and maintaining the workarounds fell to the vendors of the DNS products.

For the February 2019 flag day, there was an estimated breakage of 5.68% of all DNS servers. Two large DNS operators were responsible for 66% of this breakage. The flag day was considered a success, as the pressure generated compelled the operators to fix their systems, and no other significant breakage was reported on that day.

Motivated by the success of this first flag day, the DNS server vendors plan another in 2020. No exact date has been set at the moment. On the next flag day, new DNS software releases will change the default settings for EDNS buffer size from today's 4096 bytes to a value around 1220 bytes. The goal is to prevent fragmentation of IP packets, which is known to be broken in some networks and can be a security risk. For this change, authoritative servers and DNS resolvers must be able to operate over TCP in addition to UDP. The main problem is misconfigured firewalls that block DNS over port 53/TCP.

The flag day website will be updated with detailed information about the date and will include online tests so that DNS administrators can test their systems.

More tomorrow!

RIPE 78 is a busy event, with much more going on than we were able to report here. Do visit the session archives to check the other presentations - there are plenty more good talks to dig into. We’ll be back with more RIPE coverage tomorrow!

Topics: IPv6, IPv4, DNS, DDoS, RIPE 78, OpenINTEL, KSK roll

The RIPE-javik logs: Day 1

Posted by Carsten Strotmann on 5/21/19 6:35 AM

ripe day 1

carsten@menandmice:~$ cat ~/ripe/ripejavik-day1.txt | blog-publish

The first day of RIPE 78 started with the welcome talk by RIPE chair Hans Petter Holen and the hosts of the meeting: Icelandic sea-cable provider Farice and RHnet, the university network provider of Iceland.

It was impressive to hear that Iceland has already achieved 80% Fiber-to-the-Home installation and will have 100% by 2025. In terms of Internet speed, Iceland is only second after Norway (for mobile Internet) and Singapore (for fixed line Internet).

After short talks by Andrew Sullivan, the president and CEO of ISOC, the Internet Society (the organisation that facilitates Internet Standards Processes such as those developed by the IETF, amongst other things) and Benno Overeinder for the RIPE Program Committee, probably the best known Icelander in the Internet community took to the stage: Ólafur Guðmundsson, inventor of DNSSEC and currently CTO of Internet accelerator Cloudflare.

IPv4 volatility

Ólafur’s topic was the volatility of IP addresses. While at the beginning of the Internet IP addresses were stable over a long time and could be used to identify a machine, this is not the case today. Mobile devices switch networks constantly, always getting new addresses. A smartphone can have more than 10 different IP addresses over the course of a single day, roaming across different mobile providers and wireless networks.

As Ólafur described, IP addresses cannot reliably be used to identify machines anymore. Still, many service providers, companies, and government agencies do that all the time, for

  • blacklisting,
  • geo-location,
  • to calculate online advertising prices by placing a value to the user using the IP address,
  • or to find the nearest content server.

IPv4 address brokerage makes the situation worse. Because there are no free IPv4 addresses left and many companies have not yet switched over to IPv6, IPv4 addresses are valuable (>20 US$ per address) and are for sale. When sold, these addresses change location, but providers of location databases cannot keep up with the changes and the databases become outdated and full of wrong data.

Roaming and routing

In the next talk, Alo Safari Khatouni spoke about the implication of mobile phone roaming in Europe. In his research, he has specifically looked into how IP data is being routed in roaming situations, and when the difference in latency and bandwidth impacts a roaming user’s experience.

He found no content discrimination (i.e. that certain data is being throttled during a roaming situation), but latency was certainly higher. Mobile network operators route the roaming traffic back to the home network, where it is then routed to the Internet. This means that for a customer of an US-based mobile network operator (MNO) who is in Iceland and trying to access a website in Iceland (to look up the weather conditions - vital information in Iceland!), the network data will be routed through the US MNO’s network. It’s no surprise that this is slower than staying in Iceland and accessing the data directly.

In the Q/A session following this talk, IPv6 evangelist Jan Zorz mentioned that he also experiences IPv6 Path-MTU-Discovery issues while being inside one of the MNO networks in Iceland. It may be that possibly someone is blocking ICMPv6 on the network.

ATLAS

In the first lightning talk, Christopher Amin from the RIPE NCC explained some of the security safety belts RIPE has built into the RIPE ATLAS system.

RIPE ATLAS is a network measuring networks, where ATLAS probes are distributed all around the world. These probes can be remotely controlled by researchers to make traffic measurements on the Internet from different points of the worldwide network. However, some probes are operated by private persons in their home networks and might be located in countries where access to certain Internet content is prohibited by law. Law enforcement might not be able to tell apart access from a real Internet device from that of an ATLAS probe.

To resolve this, RIPE has built in a host of security and safety measures to limit or block the access to sensitive Internet content, but also wants to add support for DNS-over-HTTPS (DoH) measurements to the ATLAS system. The problem here is that DNS-over-HTTPS looks, by design, like HTTPS traffic generated by a web browser. From the outside, one cannot see if the content requested is a website or DNS data. Enabling DoH measurements without restrictions can introduce risks for RIPE ATLAS probe operators. Christopher asked the RIPE community about their comments and how this challenge can be solved.

The second issue Christopher brought up was the use of EDNS (Extended DNS) options in ATLAS experiments. Researchers would like to test new or unspecified option values against DNS servers on the Internet, but this can lead to unexpected behaviour, even crashing DNS servers (if the DNS server software is not of high quality, which sometimes happens if network equipment vendors write their own implementations of DNS). There’s a risk in probing these EDNS options, but Christopher is not sure exactly how big the risk is.

IPv6-only

In the last lightning talk of Day One of RIPE 78, security expert Enno Rey presented his insights from an IPv6-only WLAN study that his company ERNW has conducted for a client. They found that mobile apps, especially on Apple’s iOS "just work.” (Which is no big surprise, as each app is tested by Apple to make sure it works as expected in an IPv6-only environment.)

ERNW found some applications that did not work out of the box and needed manual fixes, like the popular game "Fortnite" and its associated Epic Game Launcher. An XMPP (Jabber) component in the game only asked for IPv4 addresses (and the domain name has no IPv6 AAAA addresses), so this was naturally failing in a network without IPv4. Some other applications like Discord worked, but had some loss of functionality.

More tomorrow

This concludes our first report from RIPE 78. Check out our guide to both the event and the city, and stay tuned for more tomorrow.

Topics: IPv6, DNS, RIPE 78, ATLAS

Why follow Men & Mice?

The Men & Mice blog publishes educational, informational, as well as product-related material for everyone and anyone interested in IP Address Management, DNS, DHCP, IPv6, DNSSEC and more.

Subscribe to Email Updates

Recent Posts

Posts by Topic

see all