The Men & Mice Blog

Everything’s changed in DNS. Nothing’s different in DNS.

Posted by Greg Fazekas on 5/10/18 7:56 AM

globe_menmicelogo

The history of DNS (Domain Name System) starts with the earliest of early networked systems: ARPANET. DNS has often been characterized as the “phone book” for the internet — that analogy was, of course, invented  in an era where phonebooks were a thing.

It may be more fitting to liken it to a phone company switchboard. Even in the earliest days of ARPANET, however, the required communication was to send an email to Stanford Research Institute at Stanford University, where the hosts.txt file was maintained, to get a new Hostname into the list. Then, all Internet hosts updated the hosts.txt file twice a week via FTP file transfer. Twice a week... by FTP!

Interesting fact: whatever platform you’re using, chances are you can find a hostsfile somewhere on your computer. This is a remnant of the early ARPANET days. Back then, a simple static text file controlled the entirety of the network.

The Making of a Network (a.k.a. “everything has changed”)

As the networks grew, the need to wait became cumbersome, if not unbearable. Business was increasingly conducted outside of bank hours, and computers were moving data faster for us. So, why not use computing to handle IP assignments as well?

In 1983, the standard for DNS was accepted by the ARPANET community. By 1984, at UC Berkeley, we saw the advent of "Open Source" Berkeley Unix Distribution (BSD) and ported TCP/IP to Unix, making Unix a networked OS (under a DARPA grant) resulting in the first version of the Berkeley Internet Name Domain (BIND). To this day BIND serves as the de facto DNS software of the internet.

Thereafter, the Internet Engineering Task Force (IETF) was founded, and with it came new formal processes that have shaped the backend of the internet as we know it today.

Fast Forward To Now (a.k.a. “nothing’s different”)

You may be thinking: that’s interesting and all, but what does that have to do with my DNS network?

Well:  DNS hasn’t changed much in the last 4 decades. Of course, the explosive growth of the internet has changed the ways we map, scale and secure our networks. But the fundamental operating principles of DNS haven’t changed since its inception-- it’s still the switchboard of the internet. Instead of humans making a request through email, however, systems can call on DNS services, any time of day, to assign a multitude of IP addresses at a time.

The introduction of IPv6 (which has yet to be fully realized), and the dawn of cloud computing and IoT (Internet of Things) brought  significantly increased device requests and IP traffic.However, all that has not changed what DNS does; rather only how it does it.

We’ve Seen It Change and Stay the Same

DNS has changed very little; but the way we utilize it changed immensely. We've seen it: since the 1990’s, Men & Mice has serviced enterprise companies with DNS, DHCP and IPAM solutions.

We proactively evolve our overlay network management solutions to meet the needs of enterprise customers, and now high growth IoT companies as well. (Note: the two are not mutually exclusive.)

We are working with perhaps the most fundamental building (scaling) block of the internet. Our expertise is focused on the importance of adaptation. Network infrastructures have become hybrid, or have moved to the cloud completely. Multitudes of DNS services and environments have come to market introducing greater choices, but also complexities for network managers.

Men & Mice and the future of DNS

menandmice_unified_console_

Men & Mice has evolved its DNS, DHCP and IPAM solutions to cater to these changing environments. We adapted to become more flexible, so that the networks of our clients can migrate across network vendors easier. We created a  unified network management console to manage, in one place, all of the diverse platforms that make up a company’s network.

We’ve introduced new services such as xDNS in 2017 to help companies manage all external DNS. Likewise, we added deeper functionality with Microsoft Azure and Azure DNS for Microsoft customers with large domain portfolios.

We’ve streamlined our sales and customer journey processes, to reflect the same ease of use customers experience in our software solutions. Get a Live Demo directly from our website, for example.

We continue to offer some of the most sought-after training courses for companies and individuals who wish to learn or sharpen their understanding of DNS, enabling them to significantly increase expertise levels across their teams.

Meet our team

Join us in Berlin on May 15th, for a special event with the Embassy of Iceland in Germany. We will discuss the “State of Network Management” and the new challenges of DNS, DHCP and IPAM.

Or, meet us at Managed Service Hosting Summit, Cisco Live, Microsoft Inspire,VMWorld and Microsoft Ignite in the coming months.

Interesting fact: Bob Metcalfe, who invented the standard of the ethernet, predicted in 1995 that the internet would collapse in a year. He  also envisioned an end to wireless technologies, and that computers would stay wired.To his credit, he did — as per his promise — eat his words, literally, after none of those things happened.

menandmice_dns_CLUS_ linkedin

Topics: TechEd, DDI, DNS

Thinking of doing DNS better?

Posted by Men & Mice on 3/20/18 10:27 AM

I train, therefore I am

Or that’s what Descartes may have said if he’d been thinking his thoughts in 2018.

Mind you, this blog is not about thinking and it’s not about physical training either, like running or wife carrying or stuff like that. It’s more about training as in training for the mind. Learning useful things. Like how to configure BIND, debug DNS, figure out TSIGs or what DNSSEC can do for your network. Basically, the kind of training that helps you build a leaner, stronger, fitter network, and create the system resilience needed to deliver those constantly surging numbers of packets to their right destination, faster and more securely.

DNS sync.png

Getting DNS skills in sync

Since 1999, Men & Mice has been known for running effective and efficient DNS & BIND training courses worldwide. Previous offerings included open, public courses in a number of locations, as well as private on-site training on request.

Beginning in 2018, we are putting a little extra effort and logging a few more air-miles, making it much easier for you to attend, wherever you are.  We’re extending our public offerings into new destinations, with upcoming courses scheduled in California, New York, Switzerland, England and Israel, with additional courses to be added as the year progresses. See the schedule at menandmice.com/training/

To get the hang of running a better network, sign up for the 3-day DNS & BIND Fundamentals, or take our most popular course and spend 5 days sinking your teeth deeper into the subject matter in DNS & BIND Week. A range of on-site training options is also on offer.

Reach out to Men & Mice Training to register for a course, ask questions, log comments, or to recommend additional locations for future public offerings.

In the meantime, check out the dates and feast your eyes on the list of topics covered by our hands-on courses, taught by DNS experts.

Happy training!

Topics: Men & Mice, DNS, BIND, DNS training

Secure Your DNS Across Multiple DNS Service Platforms with Men & Mice xDNS Redundancy

Posted by Men & Mice on 7/10/17 12:50 PM

DNS (Domain Name System) is the most critical aspect of any network’s availability. When DNS services are halted, or slowed down significantly, networks become inaccessible, leading to damaging losses in revenue and reputation for enterprises.

To ensure optimal network availability, many enterprises depend on top-tier managed DNS service providers for their external DNS needs. The basic “table stakes” characteristics of an enterprise-class managed DNS service are high reliability, high availability, high performance and traffic management. However, even the most robust DNS infrastructure is not immune to outages.

Outages may be localized, in which only certain DNS servers in the network are not responding, or, less commonly, system-wide. A system-wide DNS failure can take an entire business offline - the equivalent of power failure in every one of their data centers.

To prevent this, top-tier managed DNS systems have a great deal of built-in redundancy and fault tolerance, yet the danger of a single point of failure remains for enterprises that rely solely on a single-source DNS service.

If no system of DNS is failure proof, this begs the question: what should an enterprise do about it?

Using multiple DNS service providers for ultimate DNS redundancy

DNS availability statistics for managed DNS providers shows that the industry norm exceeds 5 nines (99.999%) uptime. This is the equivalent of about 5 minutes per year downtime. However, this top line number does not provide any detail on the impact of degraded performance, or the cascading effect of a system-wide outage of various duration, on individual enterprises.

To discover the true impact of a potential loss of DNS availability, enterprises need to properly assess the business risk associated with relying on a sole source provider, and compare that with the cost of a second source DNS service. What would a 30-minute loss of DNS cost the business in terms of revenue loss, reputation damage, support costs and recovery? What does it cost to maintain a second source DNS service?

Research amongst enterprises for whom online services are mission critical generally concludes that the cost ratios are in the range of 10:1 – one order of magnitude. Put another way, the cost of one outage is roughly estimated to be ten times the annual cost of a maintaining a second service. A business would have to have second source DNS for ten years to equal the cost of one major DNS outage.

Looking at the odds and costs of outages, many enterprises are opting to bring in a second, or even a third, DNS service to hold copies of critical DNS master zones.

This system of external DNS redundancy boosts DNS availability by:

External-DNS-Redundancy.png

1. removing the danger of exposure to a single point of DNS failure.

2. reducing traditional master-slave DNS redundancy vulnerabilities, where slave zones can’t be changed if the master becomes unavailable.

3. improving infrastructure resilience by hosting critical zones with multiple providers, ensuring continued service availability and updates of changes if one DNS service provider becomes unavailable.

The risky business of maintaining DNS redundancy across platforms

In theory, DNS redundancy across multiple DNS service provider platforms should be the best solution for optimal DNS high reliability, high availability and high performance. In practice, however, the complexity of tasks and scope for error involved in replicating and maintaining identical DNS zones on multiple platforms pose additional threats to DNS availability. The situation is made worse by:

  • A lack of centralized views
  • A lack of workflow automation
  • The difficulty of coordinating multiple platform APIs

This inability to view, synchronize and update identical zones’ data simultaneously can, in itself, lead to errors and conflicts in DNS configuration and result in a degradation of network performance, or even a network outage – the very events that multi-provider DNS redundancy is intended to prevent.

Protect your DNS on multiple platforms with Men & Mice xDNS Redundancy

Breaking new ground in the battle against DNS disruption, the Men & Mice xDNS Redundancy feature provides the abstraction level necessary to replicate and synchronize critical DNS master zones across multiple DNS service provider platforms, on-premises, in the cloud, or in hybrid or multi-cloud environments.

Men & Mice xDNS provides a unified view and centralized management of DNS data, regardless of the DNS service provider platform. Network administrators and other authorized users can use xDNS to perform necessary updates to their network’s DNS, as well as benefit from building automation with the powerful Men & Mice API, instead of having to dig around in different DNS platforms and deal with coordinating conflicting APIs. DNS-redundancy-and-Men-and-mice-suite.png

Combined with the flexibility of building automation on top of the Men & Mice Suite, xDNS offers you the freedom to better distribute your DNS load based on zone priority, performance requirements and accompanying costs. With xDNS, you are better equipped to steer the tiered price points of externally hosting, for example, critical high-performance or less essential low-performance zones, and utilize the DNS service best suited to your situation at a given time.

 


How xDNS Redundancy Works

Using the Men & Mice xDNS feature, create a zone redundancy group by selecting critical zones from DNS servers and services such as BIND, Windows DNS, Azure DNS, Amazon Route 53, NS1, Dyn and Akamai Fast DNS.

Once an xDNS zone redundancy group has been created, xDNS assists the administrator in creating identically replicated zone content, resulting in multiple identical master zones. Additional zones can be added or removed from the xDNS group as required.

All changes initiated by the user through Men & Mice, both the UI and API, will be applied to all zone instances in the group. All changes made externally to zones existing in the xDNS group, will be synchronized to all zones in that particular xDNS group. However, if DNS record conflicts arise, xDNS will alert the user and provide an option on how to resolve conflicts before the group is re-synchronized.

If an xDNS zone is not available for updating, for instance if one DNS service provider experiences an outage, that zone will be marked as out-of-sync. Once the zone becomes available again, it will be automatically re-synchronized and will receive all updates that were made while the DNS service was unavailable.

Men & Mice and NS1

NS1, the leading intelligent DNS and traffic management provider, recognizes the growing need for diverse application resiliency. NS1 has joined forces with Men & Mice in improving the efficacy of external DNS redundancy. Kris Beevers, Co-founder and CEO, says:

"Leveraging multiple managed DNS networks is the clear best practice for maintaining 100% uptime in today's rapidly evolving operational environment.  Configuring and operating multiple managed DNS services can be a complex, time-consuming process.  NS1 is excited to partner with Men & Mice to help enterprises minimize management overhead and seamlessly enable redundant DNS. xDNS Redundancy is well-suited to enable multi-network DNS without the usual headaches."

Men & Mice xDNS – making external DNS redundancy truly resilient

DNS redundancy is a great concept on paper, but a daunting challenge in practice. With xDNS, enterprises can seek out second, or even third source DNS services, confident in the knowledge that their DNS, and ultimately their business, will truly be safer that way.

Magnus Bjornsson, Men & Mice CEO, considers xDNS an important step towards providing enterprises with greater, and more reliable, network availability.
“Recent prominent network outages once again illustrate the critical importance of building more effective network resiliency through a powerful and secure system of DNS redundancy. Men & Mice xDNS provides a simple way for companies to manage their DNS on multiple external platforms, with the Men & Mice Suite software automatically taking care of the replication and synchronization of data in a reliable and consistent manner. We are looking forward to cooperating with NS1 on developing xDNS and extending DNS redundancy offerings.”

Men & Mice xDNS takes the ‘daunt’ out of maintaining external DNS redundancy, providing the centralized views and control necessary to reduce the risk of network exposure to a single point of failure, improve network reliability and performance and bolster the successful mitigation of DDoS attacks and other potentially harmful DNS incidents.

To learn more about xDNS Redundancy, check out the xDNS webinar, jointly presented by Men & Mice and NS1.

Check out the video to discover how it DDI all comes together:

Or try it out in the Men & Mice Suite:

New Call-to-action

Topics: DNS, Security, High availability, DNS redundancy, DDoS, External DNS, Failover

Men & Mice Breaks New DDI Ground with xDNS Redundancy and Multi-Cloud IPAM

Posted by Men & Mice on 6/29/17 1:30 PM

The joke goes: “How did God create the universe in seven days? No legacy infrastructure.”

Funny (or not) as that may be, how to make the most of legacy infrastructure in the age of accelerating technological disruption and rapid cloud services adoption, is the harsh reality most enterprises face today.

Well-known for its fast, reliable and efficient performance on large enterprise networks, the Men & Mice Suite already has a reputation as the go-to, enterprise-class, software overlay DNS, DHCP and IP Address Management (DDI) solution. With the release of Version 8.2 of the Suite, Men & Mice further solidifies our position as the commercial DDI solution best equipped to help large enterprises capitalize on legacy infrastructure, while adopting cloud services to advance business agility and scalability.

The Men & Mice Suite – IP wherever you are 

architecture.png

Almost three decades of expert innovation in DNS, DHCP and IP Address Management has given Men & Mice unique insight and expertise into creating solutions that confidently mitigate the shocks of technological disruption.

Built as an enterprise-grade, back-end agnostic solution and deployed on top of DNS and DHCP infrastructure, the Men & Mice DDI Suite pulls together critical network data from wherever it is kept, on-premises, in the cloud, hybrid cloud or multi-cloud, and turns a potential hot mess into a comprehensive overview, accessed and controlled from a single pane of glass.

The Men & Mice Suite provides consistent administrative controls on heterogeneous networks, with unparalleled support for Windows DNS and DHCP, BIND, Unbound, PowerDNS, ISC DHCP, Kea DHCP, Cisco IOS, OpenStack and Azure DNS and Amazon Route 53.

Designed to integrate seamlessly with the VMware Orchestrator framework, the Men & Mice Suite VMware vRealize Orchestrator plug-in allows for fast and efficient provisioning of virtual machines.

The first DDI solution to fully integrate with Microsoft Active Directory (AD), the Men & Mice Suite incorporates management of users and groups through AD, while granting access rights and building up roles and responsibilities through the Men & Mice Suite, ensuring advanced and secure granular role-based access management.

Offering you the flexibility to control your network as it suits you best, the Men & Mice Suite provides three powerful interfaces: the Men & Mice management console, the Men & Mice web interface, and, the strong and consistent Men & Mice API, communicating in SOAP, JSON-RPC and REST. The Men & Mice API, especially popular with our customers, provides the robust abstraction tools necessary to build and extend automation.

New in Men & Mice Suite Version 8.2

From Version 8.2, the Men & Mice Suite’s back-end agnostic capabilities are extended to include advanced, multi-cloud IP Address Management and integrated support for external DNS service providers.

Building on the flexibility of its architecture, Men & Mice Suite Version 8.2 consolidates on-premises and cloud networks in one view and point of access through support for IPAM in Azure and AWS, and by adding support for DNS service providers NS1 and Dyn to existing Men & Mice support for Azure DNS and Amazon Route 53.

Unique on the DDI market, and new in Version 8.2, the Men & Mice xDNS redundancy feature enables multi-platform DNS redundancy for ultimate network high availability, and successful mitigation of the fallout from DDoS attacks and other DNS failures.

xDNS redundancy provides the abstraction level necessary to replicate and synchronize critical DNS zones across multiple DNS service provider platforms, eliminating the possibility of a single point of failure resulting from dependency on one external DNS service provider.

Men & Mice - Changing the way the world sees networks

As IT matures into a key element for easily scalable business development and product delivery, and ultimately a driver of business growth, the need for high network availability, reliability and performance escalates.

For Magnus Bjornsson, Men & Mice CEO, delivering DDI products that boost business performance by bridging the gap between on-premises, cloud, hybrid cloud and multi-cloud network environments, is a challenge happily accepted. “We live in a world that’s getting more complicated by the minute. Cloud vendors are continuously bringing powerful new services online and enterprises are wrestling with how and when to best utilize them. Men & Mice Suite Version 8.2 is a landmark release, tackling this great challenge with innovative new features. Consolidating hybrid and multi-cloud IP Address Management in a single view and bolstering DNS availability across service provider platforms with xDNS redundancy, are great steps towards strategically improving the most critical of a company’s IT assets – its network. The Men & Mice Suite, used to run some of the largest corporate networks on the planet, is designed to give you the freedom and flexibility to use the back-end platform you want, to build the network you need.”

Looking for more?

Follow these links for more information on Men & Mice xDNS redundancy feature, or multi-cloud IP Address Management.

To see Men & Mice xDNS redundancy in action, check out the xDNS Redundancy webinar, jointly presented by Men & Mice and NS1.

Curious about how the Men & Mice Suite can benefit your network? Get in touch with one of our Men & Mice Sales Engineersor get your free Version 8.2 license for a complimentary 30-day trial experience.

New Call-to-action

Topics: IPAM, DNS, Security, CLOUD, High availability, DNS redundancy

Ready for another look at DNSSEC?

Posted by Men & Mice on 4/12/17 8:32 AM

dnssec.pngSince the dawn of DNS, it has been a system regularly experiencing phases of increased vulnerability. Yet never before has it been as vulnerable to the escalating size of DNS attacks as in recent years, most notably in 2016.

Advice on how to prevent, or at least mitigate, all manner of attacks on DNS proliferates, and every security vendor and his uncle promises heaven and earth, if only you bought into their solutions. While you should investigate all options and carefully devise a wholescale security strategy, together with overhauling your network’s architecture design to close unnecessary gaps and eliminate weak links, it is critical that you don’t leave one of the most obvious DNS security stones unturned – DNSSEC. 

After Dyn went down so spectacularly last October during the biggest DDoS attack recorded to date, Geoff Huston gave an excellent talk at RIPE 73, speculating on possible ways to mitigate DNS attacks. In the process, he also managed to remind the audience that one of the ways to make DNS (and conversely, the internet) safer would be to fully implement DNSSEC. Fully deployed, DNSSEC ensures that the end user is connecting to the intended, and verified, website or service corresponding to a specific domain name. In this way, DNSSEC protects the directory lookup and complements other security technologies, such as TLS (https:). DNSSEC is not a magic bullet and won’t solve all internet security issues, but in a world of constantly multiplying mutations of attacks on DNS availability, it sure can’t hurt to add it to your DNS security repertoire.

That said, DNSSEC would be a much happier prospect for most of us if it were not so tedious to set up. Still, like all things worthwhile, a little bit of initial effort can take you a long way. To help you get a grip on the ins and outs of DNSSEC, Men & Mice’s DNS expert Carsten Strotmann recently added a DNSSEC zone signing tutorial to our useful selection of DNSSEC resources, all bound to help you take steps towards DNSSEC with greater confidence. The DNSSEC zone signing tutorial follows on from Carsten’s highly rated November 2016 webinar on DNS and DNSSEC monitoring – Strategy and Tools. An added bonus is the scripts of 15 essential DNS and DNSSEC monitoring tests which can come in pretty handy once you’ve set the DNSSEC wheels in motion.

In the greater scheme of dealing with DNS vulnerabilities, it’s reassuring to know that organizations such as the IETF are dedicated to coming up with solutions to better protect the internet at the top levels of design. The DNS PRIVate Exchange Working Group (DPRIVE – a simply brilliant acronym, as they go) is tasked with developing mechanisms to enable the confidentiality of DNS transactions. While DNSSEC revolves around ensuring that data remains unchanged during communication, the data itself remains open, so to speak. DPRIVE is working towards concealing the data, primarily focusing on providing confidentiality between DNS Clients and Iterative Resolvers, but perhaps later on progressing towards providing end-to-end confidentiality of DNS transactions. In practice, these developments mean that somewhere down the road, it will hopefully be possible to:

  1. provide DNS servers with knowledge on how the structure of the internet works so DNS queries will have a straighter and narrower path, only asking for the data that is really required and not having to put in full requests that have to go all the way to the root name servers.

  2. encrypt communication between the DNS resolver (usually on the internet provider’s network) and authoritative servers on the internet so that data transmitted can’t be harvested by ill-intentioned entities.

One of the side benefits of this type of encryption is that the underlying transport protocol will likely switch from UDP to TCP, thereby providing the ‘handshake’ required for secure communication and making spoofing so resource intensive that it will take the easy fun out of the kind of DoS attacks we’ve seen escalating in recent years.  

With all new and generic top level domains, as well as country code top level domains DNSSEC signed today, the implementation of DNSSEC to make the internet more robust and secure is quickly turning into the rule, rather than the exception. Which begs the question: why wait till tomorrow when you can begin implementing DNSSEC on your domain today?

Free trial of the Men & Mice Suite

 

Topics: DNSSEC, DNS, DANE

Microsoft Azure DNS and Men & Mice Making More Sparks Together

Posted by Men & Mice on 9/29/16 10:57 PM

Chemistry. Sometimes, when two separate entities meet, they just have it. Sometimes they don’t. When it comes to Men & Mice and Microsoft, it’s definitely a case of the former. There’s surefire chemistry, and, even though the relationship already dates back to way back when, we’ve never been stronger together than we are now.

Just this week, Microsoft Azure announced General Availability of their domain hosting service, Azure DNS, in a joint statement with Men & Mice, released on September 26th at the Microsoft Ignite conference in Atlanta, USA. The General Availability announcement comes slightly more than a year after Microsoft first unveiled the public preview of this new addition to their cloud network offerings at Microsoft Ignite in Chicago in 2015. Men & Mice had already announced support for Microsoft Azure DNS in January of this year, with the release of the Men & Mice Suite Version 7.1. Now everyone is able to pick the fully ripe fruits of this productive partnership.

According to Jonathan Tuliani, Program Manager for Azure Networking – DNS and Traffic Manager, with this announcement Azure DNS is now ready to be used for production workloads. Given that Azure DNS “is supported via Azure Support and is backed by a 99.99% availability SLA” this means that Men & Mice Suite customers can now sit back and enjoy the high availability, performance, low cost and convenience of hosting their domains in the cloud with Azure DNS, while maintaining full control of their DNS domains and IP address blocks with the help of the powerful DNS, DHCP and IP Address Management (DDI) tools provided by the Men & Mice Suite.

azure_dns.jpg

Magnus E. Bjornsson, CEO of Men & Mice, sees this as one more positive step towards the continued development of third-party support products in close cooperation with Microsoft. “We are proud partners of Microsoft and embrace the opportunity to join forces with this leader in the field of IT. Our mutual collaboration enhances the value of the open and adaptable Men & Mice Suite to our customers.”

The Men & Mice Suite already exhibits a core, unfettered synergy with Microsoft Active Directory, which helps to make it one of the world’s top choice suppliers of DNS, DHCP and IP Address Management software solutions. With the addition of support for Azure DNS, as well as support for Windows Server 2016, there’s no telling where this juicing up of existing chemistry will take us next. If the past is anything to go by, it’s bound to be a happy combination of small steps and giant leaps towards collaborative, innovative creation.

The full General Availability announcement can be accessed on the Microsoft Azure blog.

 

Topics: Men & Mice Suite, DNS

RIPE 72 – A Blog Report on DNS & IPv6

Posted by Carsten Strotmann on 6/22/16 5:30 AM

RIPE 72 took place in Copenhagen from 23-27 May 2016. This blog report shares some of my thoughts on interesting talks and presentations on DNS and IPv6.  

As always, this report cannot be exhaustive and I recommend that those interested browse the meeting archive of RIPE 72 for other interesting topics.

DNS

Victoria Risk from ISC reported on the changes in the upcoming BIND 9.11 release (BIND 9.11 Release Update) that is planned for August 2016. The new catalog zone feature allows automatic provisioning of slave zones from a central catalog zone. New zones are configured as a master zone on one server and a special entry is written into the catalog zone, a meta-data zone that is configured on the master and all secondary servers. The catalog zone will be replicated by zone-transfer and the secondary server will automatically configure a slave-zone for the newly added domain.

Men & Mice Trainer Jan-Piet Mens has already had a chance to test this new feature and wrote a blog article about it: Catalog zones in BIND 9.11. ISC has issued an Internet Draft in the IETF about catalog zones with the hope that other DNS software vendors will implement a compatible version.

BIND 9.11 will include a new, refined backend for storing DNS zone data in databases, called the dyndb api. This new API is much faster than the older DLZ API and also works with DNSSEC.

Speaking of DNSSEC, BIND 9.11 will come with a new component called dnssec-keymgr that will be able to automate DNSSEC key-rollover based on a policy, much like the external OpenDNSSEC tool. More improvements to BIND 9.11 can be found in the presentation and also in the upcoming Men & Mice Webinar What's new in BIND 9.11.

Jeff Osborn from ISC started a discussion on a license change of the BIND 9 DNS Server in his talk Changing the Open Source License on BIND. Today, the BIND 9 DNS server is licensed under the ISC license, which is a permissive BSD-style license. Jeff proposes a switch to the Mozilla Public License (MPL), which is a so-called copy-left license. Both licenses are open-source licenses, but the main difference is that the MPL requires all source code changes to the product to be made public. This license change will have no negative effect on anyone using the BIND 9 DNS server, but might affect companies that build products that incorporate the BIND 9 server code. As an overlay management solution, the Men & Mice Suite product works with an un-altered BIND 9, so customers using the Men & Mice Suite would also not be affected by such a license change. Jeff welcomes any feedback on the license change. His contact information can be found in the talk's slides, available in the link above.

Patrik Faltstrom, Chair of the Security and Stability Advisory Committee on the DNS root-server system, presented an alert on WPAD Name Collision Vulnerability. WPAD, the "Web Proxy Auto-Discovery", is a way to configure the Web-Proxy to be used by a Web-Browser using DNS. In this function, the special domain name "wpad" is resolved in the local domain name of the network the client is in. Collisions with internal, non-registered domain names and new top level domains in the Internet DNS system now create the vulnerability that external parties can control the internal proxy configuration inside a company's network. Internet Explorer on Windows systems have this function enabled by default, but it can also be enabled in Firefox, Safari or Chrome-Browsers on MacOS X, BSD and Linux. Running an unregistered TLD in an internal DNS deployment is not recommended, but DNS administrators will find it difficult to remove the sins of the past. Administrators should block DNS queries for internal-only domains at their DNS-resolvers, monitor DNS queries leaving the network for internal names and consider manually switching off the WPAD function in the browsers.

Duane Wessels from Verisign gave a talk on the size increase of the Root-Zone Zone-Signing-Key (ZSK). Since the beginning of the DNSSEC-signed root-zone, the ZSK was a 1024bit RSA key, as recommended by RFC 6781 - DNSSEC Operational Practices, Version 2. However, while not an immediate security threat, 1024bit RSA keys are now also seen as having a too small security margin when used for DNSSEC signatures (1024bit RSA keys have been too weak for encryption for many years). The new ZSK will be a 2048bit key and it will be introduced into the DNS root-zone on 20th September 2016. All testing done so far indicates that there should be no problems. Even though the DNS responses from the root zone during a ZSK rollover do increase from 883 to 1138 octets/bytes, the response is still below the 1232byte EDNS0 limit often used in the IPv6-DNS-Resolver or the 1500byte Ethernet MTU.

The Unbound DNS-Resolver now implements DNS Query Name Minimisation to Improve Privacy, RFC7816. Ralf Dolmans of NLnetLabs explains in his talk QNAME Minimization in Unbound how this new feature is implemented. In traditional DNS, a DNS resolver always asks the full question to all servers in the delegation chain. This is because the DNS resolver does not know about the delegation topology of the DNS system in use. In the Internet, there is a defined delegation structure for DNS, starting with the root-zone, the generic, new and country-code top-level-domains and second-level domains owned by companies and individuals below it. In the Internet, a DNS-resolver can shorten the query when asking at the root-zone or TLD level, enhancing the privacy of the users of the DNS resolver. QNAME minimization in the DNS resolver used by a client machine can be tested with a DNS lookup tool such as dig

 

% dig txt qnamemintest.internet.nl +short


IPv6

John Jason Brzozowski from US cable giant Comcast presented IPv6 @Comcast – Then, Now and Tomorrow about the challenges and successes in their deployment of IPv6 "large scale". Overall, IPv6 at Comcast is a success and they are now putting in motion the plan to phase out IPv4.

In the IPv6-Working-Group session, John reported on Community WiFi and IPv6 and how Comcast is using IPv6 to create public WIFI hotspots on CPE devices. Comcast is giving out a full "/64" network to every WIFI-device connected, in order to create easy network isolation and to reduce the multicast traffic over WLAN. This scheme could have even more benefits, such as assigning an IPv6 address for every service running on a host.

The Google public DNS resolver now supports DNS64-translation (currently in Beta) on the public DNS-resolver address "2001:4860:4860::6464" (IPv6-Only Has Never Been So Easy). DNS64 is a translation technology that works together with NAT64 to allow a client on an IPv6-only network to connect to IPv4-only services on the Internet. As DNS64 "re-writes" DNS content, it clashes with DNSSEC, as Jen Linkova from Google explains in IPv6-only and DNS(SEC|64).The workaround proposed in the talk got some criticism from the audience.

Enno Rey from the security company ERNW had a close look at the security issues of Multicast Listener Discovery MLD, a topic that has not seen much attention so far. He and his colleagues have found several issues that can be used for denial of service attacks or traffic redirection attacks by an intruder inside the local network. He recommends an (still to be developed) "MLD guard" function in switches (similar to DHCPv4- or RA-Guard) or to deploy port based ACL filtering of MLD traffic. Nobody should panic because of these findings, but every IPv6 network admin should know about MLD and the implications of having MLD active in their networks.

Vaibhav Bajpai had an interesting talk on Measuring Webpage Similarity from Dual-Stacked Hosts, looking at the differences in website content between a page fetched via IPv6 vs. IPv4. Differences coming from certain objects on the page (CSS, JavaScript, Advertisements …) are only available for one protocol, while the general website is dual-stacked and therefore available on both IPv4 and IPv6.

Two talks covered the topic of IPv6-only networks, but from very different angles. In How to Make Trouble for Yourself - You Build an IPv6-Only Network in 2016, Roger Jørgensen from Bredbandsfylket Troms in Norway reported on their project to build a new fiber optic network in the far north of Norway. The management part of this network is designed and operated as an IPv6-only network. Luuk Hendriks gave a report of his attempt at Going IPv6-only at Home while keeping the most important user of his home network, his girlfriend, happy.

Enno Rey talks about Real Life Use Cases and Challenges When Implementing Link-local Addressing Only Networks as of RFC 7404 from his experiences implementing a Link-Local-only addressing scheme in a larger enterprise network. The Link-Local-only addressing was chosen to simplify address management, as almost 50% of networks in this customer's environment are point-to-point links. There are still issues with vendor support in network devices when implementing Link-Local-only addressing. In the discussion following the talk, the audience gave a mixed message, with some people claiming success at running a Link-Local-only network.

Other topics

Mircea Ulinic presented a way to automate the provisioning and management of network devices (router, switches etc) using the configuration orchestration tool "SaltStack". SaltStack is usually used to automate the provisioning of server machines running a Salt-Agent (Minion). As it is difficult to install a customer agent on network gear, this talk presented a way to use proxy machines that act as the minions for network hardware. SaltStack automation can save a great deal of time when used in large deployments. Details can be found in Mircea's talk: Network Automation with Salt and NAPALM.

Shane Kerr, who we recently had as an interview guest in our latest Webinar on Yeti-DNS, gave a humorous talk about the "Internet of Things (IoT)" in IoT: What is the Problem or “How To Explain To Your Boss That IoT Won't Make the Company Rich….”.

Those of you hungry for more on RIPE 72, all the above talks and more can be found in the meeting archive of RIPE 72.

Topics: IPv6, DNSSEC, DNS

An Introduction to the Men & Mice Web Service: JSON-RPC

Posted by Men & Mice on 2/23/15 10:48 AM

The Men & Mice Suite is a comprehensive DDI solution that allows you to manage your DNS, DHCP and IP Addresses in a simplified, flexible manner. In order to maximize this flexibility the Men & Mice Suite provides a rich set of commands accessible through its web service interface. These commands are used internally for almost all requests between the user interface and the Central service, so if something can be done in the user interface it can also be done through the web service interface.

Initially the web service interface provided a SOAP API and a WSDL service. Starting with version 6.6, the Men & Mice Web Service also provides a JSON-RPC service that complies with the JSON-RPC 2.0 specifications. The main reason for this decision to provide JSON-RPC along with SOAP is that JSON is well supported in popular programming languages such as JavaScript and Python, and working with JSON-like data instead of XML is much easier and less error prone, not to mention that JSON packets are smaller than XML packets and more human readable.

If you are interested in the SOAP service there is an excellent introduction to it on our blog (see: Introduction to the Men & Mice Web Service API).

JSON-RPC

The JSON-RPC service supports all the same commands as the SOAP API. You can find a complete list of the SOAP commands available here. The same code is used internally in the Men & Mice Suite so when a new SOAP command is added, it will be readily available as a JSON-RPC method. As with SOAP, our main focus is to provide JSON-RPC over HTTP/HTTPS.

No special magic is needed to send a JSON-RPC request. When our web service receives a packet it will determine from the content of the packet whether this is a SOAP or a JSON-RPC request. You can even mix those two and the web service will simply respond with a JSON if this was a JSON-RPC request or an XML if this was a SOAP request. The general format of a Men & Mice Web Service JSON-RPC request and a response is as follows:

--> {"jsonrpc": "2.0", "method": "theMethodName", "params": theParams, "id": 1}
<-- {"jsonrpc": "2.0", "result": theResult, "id": 1} 

The jsonrpc member should always be "2.0". The method member should contain a string with the name of the method to be executed. The params member contains data that should be passed to the method. The member id is optional, but if it exists the response will contain the same ID as provided in the request. If successful the response will include a member result that will contain the result from the RPC method that was executed. If an error occurs the response will not include a result but instead another member named error that will include information about what happened.

An example:

--> {"jsonrpc": "2.0", "method": "Login", "params": {"password": "secretPassword", "loginName": "administrator", "server": "10.5.0.6"}, "id": 1}
<-- {"jsonrpc": "2.0", "result": {"session":"J95A5uh9obM6KjCtbtHZ"}, "id": 1}
    
<-- {"jsonrpc": "2.0", "method": "GetDNSServers", "params": {"session": "J95A5uh9obM6KjCtbtHZ"}, "id": 2}
--> {"jsonrpc": "2.0", "result": {"dnsServers":[{"ref":"{#2-#10}","name":"caching1.demo.","resolvedAddress":"10.5.0.27","port":1337,"type":"Unbound","state":"OK","customProperties":[],"subtype":"Unbound"},{"ref":"{#2-#11}","name":"caching2.demo.","resolvedAddress":"10.5.0.30","port":1337,"type":"Unbound","state":"OK","customProperties":[],"subtype":"Unbound"}],"totalResults":2}, "id": 2} 

JSON-RPC and Python

Python is powerful as a scripting language. The standard libraries are pretty comprehensive and if you need something more specific then there are plenty of additional libraries that can be found. One of the strengths of Python is its dynamic type system and "duck typing" that allows for the creation of complex objects without writing too much code.

We have created a small, light-weight Python module that uses the request library to simplify session handling and data conversion from Python dictionaries to JSON requests. The class overloads the __getattr__ method, so that all unknown functions are assumed to be RPC requests and all the arguments passed are assumed to be a RPC data.

You can find the latest version of the Python module here. To install it, extract the downloaded file, go into the subdirectory and run:

$ python setup.py install

Before we can use any of the commands we need to create a JSONClient object and log in to our Central Service. Let's start Python and type in the following commands.

$ python
>>> import mmJSONClient
>>> client = mmJSONClient.JSONClient()

The Basics

The Login method accepts the following arguments: proxy, server, username and password. Proxy is the name or IP address of a server running the Men & Mice Web Service. If you do not provide the proxy argument, then the Login method assumes the web service is running on the same machine as the Central Service. The server argument should contain the name or address of the machine that is running the Men & Mice Central Service. The username argument is the name of the user that we want to log in as and password is his password. Note that in order for this user to be able to use the web service he has to have permissions to use the web user interface. For optimal security, it is best to create a new dedicated user account that only has access to the objects the script needs to function.

Now let's log in to the server. In this example I'm logging in to a server named 'central.demo' with the user name 'a_user' and the password 'secret'. Since the web service is also running on 'central.demo' I don't need to provide the proxy argument.

>>> client.Login(server='central.demo', username='a_user', password='secret') 

As I mentioned earlier, we can use all the commands that are available for the SOAP API, and there are a lot (around 200 at last count). A list can be found here, along with descriptions of what arguments they need and what data they return.

Let's start with something simple, like the GetDNSServers method. As the name implies, the method will return all the DNS servers available to the user. Note that it might not return all the DNS servers available, but instead only those the user has permission to see. If you look at the description of GetDNSServers, you will see the only argument required is the session ID; everything else is optional. The session ID is a random string that the web service returned from the Login method. If you have successfully executed the Login method, you can see what your temporary session ID is by doing:

>>> print(client.sessionid)
VFr79UQMhFfOq4Q8l0Qi 

Or if you are using Python 2.7, use print without parentheses:

>>> print client.sessionid
VFr79UQMhFfOq4Q8l0Qi 

The rest of this article assumes use of Python 3.x. When using earlier versions of Python the syntax may have to be adjusted accordingly.

Fortunately when using the JSONClient class you don't have to worry about the session ID, as the class handles that automatically so if you want to list all the DNS servers available, you can simply type:

>>> print(client.GetDNSServers())
{u'totalResults': 2, u'dnsServers': [{u'name': u'caching1.demo.', u'resolvedAddress': u'10.5.0.27', u'ref': u'{#2-#10}', u'subtype': u'Unbound', u'state': u'OK', u'customProperties': [], u'type': u'Unbound', u'port': 1337}, {u'name': u'caching2.demo.', u'resolvedAddress': u'10.5.0.30', u'ref': u'{#2-#11}', u'subtype': u'Unbound', u'state': u'OK', u'customProperties': [], u'type': u'Unbound', u'port': 1337}]} 

The result from GetDNSServers is a Python dictionary that contains two members: totalResults and dnsServers. As stated in the SOAP API documentation, the totalResults member contains the number of DNS Servers we have access to, and the member dnsServers contains an array of DNS servers. To refer to a member in a Python dictionary we can use brackets - for example, the following can be used to print out the total number of DNS servers:

>>> print(client.GetDNSServers()['totalResults'])
2 

Providing arguments to a method is simple. GetDNSServers has a number of optional arguments, for example the filter argument. Filter is powerful argument that is provided with many of the Men & Mice SOAP API commands. It allows you to limit the result to only the items that you want to see. You can use wildcards and regular expressions with a filter. For more information, look at the description of filtering in the SOAP API. Let's say that we wanted to get all the servers that contain the number 2 in their name. We can simply do:

>>> print(client.GetDNSServers(filter='name:2'))
{u'totalResults': 1, u'dnsServers': [{u'name': u'caching2.demo.', u'resolvedAddress': u'10.5.0.30', u'ref': u'{#2-#11}', u'subtype': u'Unbound', u'state': u'OK', u'customProperties': [], u'type': u'Unbound', u'port': 1337}]} 

Or let's say we wanted to print out all A records from the zone 'applepie.ak.is' that start with the name 'apple':

>>> print( '\n'.join(['{}\t{}\t{}'.format(r['name'],r['type'],r['data']) for r in client.GetDNSRecords(dnsZoneRef= 'applepie.ak.is.', filter='type:^A$ name:^apple')['dnsRecords']]) )
apple1 A 10.50.0.1
apple2 A 10.50.0.2
apple3 A 10.50.0.3 

In the example below we have created a Python script that logs on to a server, prints out all of its zones and then the first 10 records of a single zone. It assumes that we are running Men & Mice Web Service and Men & Mice Central on a server named 'central.demo'. It also assumes there is a zone named 'applepie.ak.is' on one of the servers:

import mmJSONClient
 
client = mmJSONClient.JSONClient()
client.Login(server='central.demo.', username='a_user', password='secret')
 
result= client.GetDNSZones()
print('\nTotal number of zones: ' + str(result['totalResults']))
for zone in result['dnsZones']:
    print(zone['name'])
 
myZone = myZone= 'applepie.ak.is.'
result = client.GetDNSRecords(dnsZoneRef= myZone, limit= 10)
print('\nRecords in ' + myZone)
for record in result['dnsRecords']:
    print(record['name'] + "\t" + record['ttl'] + "\t" + record['type'] + "\t" + record['data'])

Manipulating DNS Zones

Now let's do something a little bit more complicated. Let's create a zone and add some records to it. If we look at the list of SOAP API commands we can find AddDNSZone. AddDNSZone requires at least two arguments: a session argument we don't have to worry about since JSONClient will handle that for us, and a dnsZone argument which should be of type DNSZone. The DNSZone type requires at least two members (or arguments): name and type. The name argument is a string, and the type argument is an enumeration which can be Master, Slave, Hint, Stub or Forward. If the call is successful AddDNSZone will return a reference to the newly created zone.

We also have to specify where we want to create the zone. Notice that we don't have any possible DNS server reference when creating a zone. The reason is that each DNS server can have one or more views that in turn can contain a number of zones. In most cases DNS servers will only have one view, called the default view or the empty view (''). When referring to any object in the Men & Mice Suite, you can either use its globally unique identifier (GUID), that was returned from the Central Server or you can pass on a string that will uniquely identify the object, in this case using its name (for a more detailed description on how this works, see Referencing Objects in the SOAP API). To reference a default view on a server you can simply use its fully qualified name and add a double column at the end.

As an example, let's say we want to send a reference to the default view as an argument on the DNS server dnsserver1.demo. Here we can simply pass the string 'dnsserver1.demo.:' as a DNS view reference.

>>> zoneRef= client.AddDNSZone(dnsZone={'dnsViewRef':'dnsserver1.demo.:', 'name':'test1.com', 'type':'Master'} , saveComment='A zone created using Python')
>>> print(zoneRef)
{u'ref': u'{#4-#2054}'} 

Note that we also added a save comment when creating the zone as it is sensible to always add a comment when you are making changes to the system. Also notice the return value from AddDNSZone. As before it is a Python dictionary, this time with only the one member, 'ref', that contains a reference to the newly created zone. Now let's create few records:

>>> for i in range(100,105):
...     client.AddDNSRecord(dnsRecord={'dnsZoneRef':zoneRef['ref'], 'name':'test{0}'.format(i), 'ttl':'', 'type':'A', 'data':'10.50.0.{0}'.format(i), 'enabled':1})
... 

In this example, since I'm creating multiple records, I might instead use the SOAP command AddDNSRecords. AddDNSRecords will accept multiple records and update the zone just once instead of multiple times:

>>> client.AddDNSRecords(dnsRecords= [{'dnsZoneRef':zoneRef['ref'],  'name':'test{0}'.format(i), 'ttl':'', 'type':'A', 'data':'10.50.0.{0}'.format(i), 'enabled':1} for i in range(100,105)])
{'errors': [], 'objRefs': ['{#13-#9009}', '{#13-#9010}', '{#13-#9011}', '{#13-#9012}', '{#13-#9013}']}

As a final example, let's create a script that will check if all the PTR records for a DNS server are in order. The script will read through all the forward and reverse zones and report if there are any PTR records missing or if there are any orphaned PTR records:

import mmJSONClient
 
def qualifyName(recName, domainName):
    if len(recName) == 0:
        return domainName
    elif recName[len(recName) - 1] == '.':
        return recName + domainName
    return recName + '.' + domainName
 
mmServer    = 'central.demo.'
mmUser      = 'a_user'
mmPassword  = 'secret'
dnsServer   = 'b-centos6-32.demo.'
 
try:
    print('Checking reverse zone on server [{}] ...'.format(dnsServer))
    client = mmJSONClient.JSONClient()
    client.Login(server= mmServer, username= mmUser, password= mmPassword)
    viewRef = client.GetDNSView(dnsViewRef= dnsServer + ':')
    # Get all forward and reverse zones
    zoneList = client.GetDNSZones(filter= 'dnsViewRef:' + viewRef['dnsView']['ref'])['dnsZones']
    forwardZoneList = [z for z in zoneList if (not z['name'].endswith('in-addr.arpa.') and not z['name'].endswith('ip6.arpa.'))]
    revZoneList = [z for z in zoneList if z['name'].endswith('in-addr.arpa.')]
    print('Found {} forward-zone(s) and {} reverse-zone(s)'.format(len(forwardZoneList), len(revZoneList)))
  
    # Collect all PTR records as a map of ip -> [record, zone, data] 
    # and give the user a warning if there are duplicates
    ptrRecordsMap = {}
    for z in revZoneList:
        print('Scanning reverse-zone [{}]'.format(z['name']))
        try:
            records = client.GetDNSRecords(dnsZoneRef= z['ref'], filter= 'type:^PTR$')['dnsRecords']
        except Exception as e:
            print(e)
        for r in records:
            domain = qualifyName(r['name'], z['name'])
            label = domain.split('.', 4)
            ip = '{}.{}.{}.{}'.format(label[3], label[2], label[1], label[0])
            if ip in ptrRecordsMap:
                print(  ('Warning: Ignoring reverse record [{} in {} {}] for the IP address [{}]:\n'
                        + '\talready found [{} in {} {}]').format(
                              r['name'], z['name'], r['data'], ip
                            , ptrRecordsMap[ip][0], ptrRecordsMap[ip][1], ptrRecordsMap[ip][2]))
            else:
                ptrRecordsMap[ip] = [r['name'], z['name'], r['data']]
 
    # Collect all A records as a map of ip -> [record, zone] 
    # and give the user a warning if there are duplicates
    aRecordsMap = {}
    aMatchedRecordsMap = {} # contains all record that found a match
    for z in forwardZoneList:
        print('Scanning forward-zone [{}]'.format(z['name']))
        records= []
        try:
            records = client.GetDNSRecords(dnsZoneRef= z['ref'], filter= 'type:^A$')['dnsRecords']
        except Exception as e:
            print(e)
        for r in records:
            ip = r['data']
            if ip in aRecordsMap:
                print(  ('Warning: Ignoring the A record [{} in {} {}]:\n'
                        + '\talready found [{} in {} {}]').format(
                            r['name'], z['name'], r['data']
                            , aRecordsMap[ip][0], aRecordsMap[ip][1], ip))
            elif ip in ptrRecordsMap:
                # We already have this as a PTR record but do the data match?
                if qualifyName(r['name'], z['name']) != ptrRecordsMap[ip][2]:
                    print(  ('Warning: Record [{} in {} {}] has an incorrect reverse reference:\n'
                            + '\talready found a PTR record [{} in {} {}]').format(
                                r['name'], z['name'], r['data']
                                , ptrRecordsMap[ip][0], ptrRecordsMap[ip][1], ptrRecordsMap[ip][2]))
                aMatchedRecordsMap[ip] = [r['name'], z['name']]
                del ptrRecordsMap[ip]
            else:
                # the ip wasn't found in the reverse list but it might already been matched
                if ip in aMatchedRecordsMap:
                    print(  ('Warning: Ignoring the A record [{} in {} {}]:\n'
                            + '\talready found [{} in {} {}]').format(
                                r['name'], z['name'], r['data']
                                , aMatchedRecordsMap[ip][0], aMatchedRecordsMap[ip][1], ip))
                else:
                    aRecordsMap[ip] = [r['name'], z['name']]
 
    print ('\nFound {} missing PTR-Record(s)...'.format(len(aRecordsMap)))
    for ip in aRecordsMap:
        print ('{} in {} {}'.format(aRecordsMap[ip][0], aRecordsMap[ip][1], ip))
    print ('\nFound {} orphan PTR-Record(s)...'.format(len(ptrRecordsMap)))
    for ip in ptrRecordsMap:
        print ('{} in {} {}'.format(ptrRecordsMap[ip][0], ptrRecordsMap[ip][1], ptrRecordsMap[ip][2]))
except Exception as e:
    print(e)

Summary

The primary focus of the Men & Mice development team has always been to simplify the complex task of administering a DDI environment while still being flexible, as different environments can have very different needs. To achieve this flexibility the Men & Mice Suite provides rich set of commands accessible through a web service interface.

As of version 6.6 both SOAP/WSDL and JSON-RPC services are supported. Both of these services have their advantages but JSON-RPC is more light-weight than SOAP/WSDL and a better fit for many programming languages.

Topics: Men & Mice Suite, DDI, DNS, DHCP, DNS Zone, Web Services, JSON, SCRIPTING

Hybrid Cloud DNS with the Men & Mice Suite

Posted by Men & Mice on 2/2/15 9:55 AM

Since its humble origins as QuickDNS for the Apple Macintosh, the Men & Mice Suite has evolved into a comprehensive management solution for existing heterogeneous DNS/DHCP environments, with a powerful IPAM module that allows for seamless AD Sites & Subnets management in Windows Active Directory environments. Since 2011 Men & Mice have also offered the Men & Mice Appliance for customers that desire integrated DNS/DHCP or caching appliance solutions.

Introducing the Generic DNS Server Controller

In 2014 Men & Mice introduced the Generic DNS Server Controller, an extension of the existing Men & Mice DNS Server Controller that allows the Men & Mice Suite to integrate with any DNS server product that features an API to access and update DNS data.

The result is a flexible DNS solution that can deploy and centrally manage a wide variety of DNS products within a single hybrid environment, all without having to replace existing DNS infrastructure.

Better living through scripting

With the Generic Controller, communication between the DNS Server Controller and the DNS server itself is not hard coded; instead, a scripting interface is called instead of directly accessing the server whenever changes are made through the Men & Mice Management Console (or any other Men & Mice client interfaces).

Men & Mice currently provides two scripts (written in python) to interface with a cloud based DNS service (Amazons Route53) and PowerDNS (with a MySQL back-end, which is widely used as authoritative DNS in the ISP market). These scripts can be further tweaked and configured to interface with any number of different DNS servers without changes to the core infrastructure.

Looking forward

What does this mean for Men & Mice's customers?

Put simply, it can significantly lower infrastructure costs when moving into the future; should you plan to migrate your data into a cloud based DNS service or even switch over to some other DNS product types like PowerDNS you can still use the Men & Mice Suite to manage the DNS/DHCP and IPAM with all the advantages it brings, such as audit trails, automation, and delegation.

Hybrid Cloud DNS with the Men & Mice Suite
  • An administrator requests a change on the DNS server via the Men & Mice Management Console, which connects to the Men & Mice Central server.
  • Men & Mice Central connects to the DNS Server Controller and relays the change request.
  • The DNS Server Controller deploys the change and reports success or error back to Central.
  • Central reports back to the administrator.

Topics: Men & Mice Suite, IPAM, DNS, DHCP, CLOUD

Supervising BIND 9

Posted by Men & Mice on 1/21/15 2:59 PM

BIND 9 is a mature piece of software, and compared with its predecessors BIND 4 and BIND 8 it is noticeably more stable and secure. One reason for this is the "Design by contract" programming style used by the BIND 9 team; as a result, BIND 9 is very particular about the data it consumes, and about its own internal data structures. Once BIND 9 encounters an unexpected state in its data structures, it terminates the DNS server process rather than continue running with bad data (and thus potentially compromise security).

While this behavior has clear advantages in terms of security, it can adversely affect service uptime - BIND 9 had several incidents in the past years where BIND 9 terminated because of issues inside the code or data structures, such as "BIND 9 Resolver crashes after logging an error in query.c". And for all its security benefits, an end user unable to reach Facebook may not be terribly understanding in the event of an outage.

The real issue, however, is not that BIND terminates when it comes across bad data, but rather that the process cannot automatically restart after the fact; there is no "supervisor" process in BIND 9.

Some operating systems have a built-in solution: MacOS X has launchd, and the BIND 9 version Apple delivers with the OS is automatically restarted should it terminate unexpectedly. Solaris has SMF (Service Management Facility), and BIND 9 can be integrated into SMF. Recent versions of Ubuntu, RedHat Enterprise, SuSe Enterprise, and Fedora now all use systemd, which can also monitor processes and restart them if needed.

But for Unix and Linux operating systems that do not ship with a process supervisor solution, supervisord is a strong alternative, with the added benefit of being relatively easy to install and configure. Supervisord comes as a package with many Linux distributions, and also works on BSD distributions.

The configuration below is intended for RedHat 6, but should require only minor tweaks to run on other Unix systems as well.

Installation

Supervisord is written in Python (2.4 - 2.7) and can be installed from source (where we have to download and install all dependencies) or with the help of setuptools, which takes care of downloading and installing dependencies (Meld3 and ElementTree).

Full Installation instructions can be found at [http://supervisord.org/installing.html]

Automatic Installation

‣ download "setuptools" from [https://pypi.python.org/packages/source/s/setuptools/setuptools-9.1.tar.gz]

 shell> tar xfz setuptools-9.1.tar.gz 
shell> cd setuptools-9.1 
root-shell> python setup.py install

Once setuptools have been installed, run the following command to install Supervisor and all required dependencies:

 root-shell> easy_install supervisor 

Manual Installation

Supervisor and its dependencies can also be installed manually.

‣ download "setuptools" from [https://pypi.python.org/packages/source/s/setuptools/setuptools-9.1.tar.gz]

 shell> tar xfz setuptools-9.1.tar.gz 
shell> cd setuptools-9.1 
root-shell> python setup.py install

‣ download "Meld3" from [http://www.plope.com/software/meld3/meld3-0.6.5.tar.gz]

 shell> tar xfz meld3-0.6.5.tar.gz 
shell> cd meld3-0.6.5 
root-shell> python setup.py install

‣ download "ElementTree" from [http://effbot.org/media/downloads/elementtree-1.2.6-20050316.tar.gz]

 shell> tar xfz elementtree-1.2.6-20050316.tar.gz 
shell> cd cd elementtree-1.2.6-20050316 
root-shell> python setup.py install

‣ download "Supervisor" from [https://pypi.python.org/packages/source/s/supervisor/supervisor-3.1.3.tar.gz]

 shell> tar xfz supervisor-3.1.3.tar.gz 
shell> cd supervisor-3.1.3 
root-shell> python setup.py install

Installing startscript and sysconfig

‣ download the startscript from [https://raw.githubusercontent.com/Supervisor/initscripts/master/redhat-init-jkoppe] and place it in /etc/init.d/supervisord

 root-shell> cp redhat-init-jkoppe /etc/init.d/supervisord 
root-shell> chmod +x /etc/init.d/supervisord

‣ download the 'sysconfig' file from [https://raw.githubusercontent.com/Supervisor/initscripts/master/redhat-sysconfig-jkoppe] and place it in /etc/sysconfig/supervisord

 root-shell> cp redhat-sysconfig-jkoppe /etc/sysconfig/supervisord 

Installing Bind 9 from Men & Mice repositories

‣ download the BIND 9 RPM from [http://support.menandmice.com/download/bind/linux/redhat/6.x/]<arch>/<version>/

 root-shell> yum install ISCBIND-<version>-<flavor>RHL<arch>.rpm 
root-shell> mkdir /var/named 
root-shell> useradd -d /var/named -r named 
root-shell> chown -R named: /var/named

‣ create a BIND 9 configuration file '/etc/named.conf'

options { directory "/var/named"; dnssec-validation auto; }; 

‣ create an 'rndc' configuration

root-shell> rndc-confgen -a 

‣ verify the configuration

root-shell> named-checkconf -z 

A basic configuration file for BIND 9 "named"

Below is my basic /etc/supervisord.conf configuration file for one service, the BIND 9 DNS Server:

 [unix_http_server] 
file = /tmp/supervisor.sock 
chmod = 0777 
chown= nobody:nobody 

[rpcinterface:supervisor] 
supervisor.rpcinterface_factory = supervisor.
rpcinterface:make_main_rpcinterface 

[supervisorctl] 
serverurl=unix:///tmp/supervisor.sock 
[supervisord] 
logfile = /var/log/supervisord.log 
logfile_maxbytes = 10MB 
logfile_backups=10 
loglevel = info 
pidfile = /var/run/supervisord.pid 
identifier = supervisor 
directory = /tmp 

[program:named] 
command=/usr/sbin/named -u named -f 
process_name=%(program_name)s 
numprocs=1 
directory=/var/named 
priority=100 
autostart=true 
autorestart=unexpected 
startsecs=5 
startretries=3 
exitcodes=0,2 
stopsignal=TERM 
stopwaitsecs=10 
redirect_stderr=false 
stdout_logfile=/var/log/named_supervisord.log 
stdout_logfile_maxbytes=1MB 
stdout_logfile_backups=10 
stdout_capture_maxbytes=1MB

Starting supervisord

With the configuration file in place, we can start supervisord. Make sure that BIND 9 is not started or you will end up with two instances of the BIND 9 server running, which isn't recommended. Also make sure that supervisord will be started on reboot of the server, either through a startscript or other means. Note that the supervisord packages bundled with Linux distributions install a startscript.

 root-shell> /etc/init.d/supervisord start 
root-shell> rndc status 
version: 9.9.6-P1 <id:3612d8fb> 
number of zones: 98 
debug level: 0 
xfers running: 0 
xfers deferred: 0 
soa queries in progress: 0 
query logging is OFF 
recursive clients: 0/0/1000 
tcp clients: 0/100 

server is up and running 
root-shell> ps -ef 
[...] 
root 10906 0.0 2.5 209096 12988 ? Ss 19:55 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf 
named 10908 0.7 1.6 44292 8112 ? S 19:55 0:00 /usr/sbin/named -u named -f 
root 10910 0.0 0.2 110228 1156 pts/0 R+ 19:55 0:00 ps aux 

root-shell> supervisorctl 
status 
named RUNNING pid 10908, uptime 0:03:19 
root-shell> chkconfig --add supervisord 
root-shell> chkconfig supervisord on

Great, supervisord has started, and it also started the BIND 9 process "named". DNS is working now.

Simulating a BIND 9 crash

To simulate a BIND 9 crash, we "kill" the BIND 9 named process:

root-shell> killall -9 named 

Supervisord should detect that the running BIND 9 process has terminated, and start a new one. DNS is still up and running.

Controlling supervisord

Supervisord can be controlled from the command line using the supervisorctl command. A list of all a control commands can be found with "help", and a description of each command with "help command":

 shell> supervisorctl help 
default commands (type help ): 
===================================== 
add clear fg open quit remove restart start stop update 
avail exit maintail pid reload reread shutdown status tail version 

shell> supervisorctl help status 
status Get all process status info. 
status Get status on a single process by name. 
status Get status on multiple named processes. 

shell> supervisorctl status named 
RUNNING pid 25770, uptime 0:00:12 

shell> supervisorctl stop named  named: stopped 

shell> supervisorctl start named 
named: started

Now, whenever there is a triggered assertion error in the code BIND 9 will terminate, but supervisord will bring it back from the dead. Your DNS service stays up, and your users and customers stay happy.

Read the supervisord documentation on how to setup event notifications, so that you get an e-mail notification should BIND 9 restart (should the outage be caused by a security vulnerability you might want to report it to bind9-bugs@isc.org as well).

Of course supervisord can be used to restart other processes as well, including other types of DNS Servers (NSD, Unbound, dnsmasq ...).

Topics: DNS, Linux, BIND 9, BIND, Supervisord, Red Hat