The Men & Mice IP Address Management blog has educational, informational as well as product related material, both videos and articles for everyone and anyone interested in IP Address management, DNS, DHPC, IPv6, DNSSEC and more....

RSS feed Subscribe twitter Subscribe Facebook

Subscribe via E-mail

Your email:

Try the Men & Mice Suite


Ask the Experts!

Do you have a question about DNS, DHCP, IP Address Management, DNSSEC, IPv6 or anything really?

Then go ahead, Ask our Experts! It's FREE
The best thing is, you don't have to be a client of Men & Mice to ask a question!!

The Men & Mice Blog

Current Articles | RSS Feed RSS Feed

Supervising BIND 9


BIND 9 is a mature piece of software, and compared with its predecessors BIND 4 and BIND 8 it is noticeably more stable and secure. One reason for this is the "Design by contract" programming style used by the BIND 9 team; as a result, BIND 9 is very particular about the data it consumes, and about its own internal data structures. Once BIND 9 encounters an unexpected state in its data structures, it terminates the DNS server process rather than continue running with bad data (and thus potentially compromise security).

While this behavior has clear advantages in terms of security, it can adversely affect service uptime - BIND 9 had several incidents in the past years where BIND 9 terminated because of issues inside the code or data structures, such as "BIND 9 Resolver crashes after logging an error in query.c". And for all its security benefits, an end user unable to reach Facebook may not be terribly understanding in the event of an outage.

The real issue, however, is not that BIND terminates when it comes across bad data, but rather that the process cannot automatically restart after the fact; there is no "supervisor" process in BIND 9.

Some operating systems have a built-in solution: MacOS X has launchd, and the BIND 9 version Apple delivers with the OS is automatically restarted should it terminate unexpectedly. Solaris has SMF (Service Management Facility), and BIND 9 can be integrated into SMF. Recent versions of Ubuntu, RedHat Enterprise, SuSe Enterprise, and Fedora now all use systemd, which can also monitor processes and restart them if needed.

But for Unix and Linux operating systems that do not ship with a process supervisor solution, supervisord is a strong alternative, with the added benefit of being relatively easy to install and configure. Supervisord comes as a package with many Linux distributions, and also works on BSD distributions.

The configuration below is intended for RedHat 6, but should require only minor tweaks to run on other Unix systems as well.


Supervisord is written in Python (2.4 - 2.7) and can be installed from source (where we have to download and install all dependencies) or with the help of setuptools, which takes care of downloading and installing dependencies (Meld3 and ElementTree).

Full Installation instructions can be found at []

Automatic Installation

‣ download "setuptools" from []

shell> tar xfz setuptools-9.1.tar.gz 
shell> cd setuptools-9.1 
root-shell> python install 

Once setuptools have been installed, run the following command to install Supervisor and all required dependencies:

root-shell> easy_install supervisor 

Manual Installation

Supervisor and its dependencies can also be installed manually.

‣ download "setuptools" from []

shell> tar xfz setuptools-9.1.tar.gz 
shell> cd setuptools-9.1 
root-shelL> python install 

‣ download "Meld3" from []

shell> tar xfz meld3-0.6.5.tar.gz 
shell> cd meld3-0.6.5 
root-shell> python install 

‣ download "ElementTree" from []

shell> tar xfz elementtree-1.2.6-20050316.tar.gz 
shell> cd cd elementtree-1.2.6-20050316 
root-shell> python install 

‣ download "Supervisor" from []

shell> tar xfz supervisor-3.1.3.tar.gz 
shell> cd supervisor-3.1.3 
root-shell> python install 

Installing startscript and sysconfig

‣ download the startscript from [] and place it in /etc/init.d/supervisord

root-shell> cp redhat-init-jkoppe /etc/init.d/supervisord 
root-shell> chmod +x /etc/init.d/supervisord 

‣ download the 'sysconfig' file from [] and place it in /etc/sysconfig/supervisord

root-shell> cp redhat-sysconfig-jkoppe /etc/sysconfig/supervisord 

Installing Bind 9 from Men & Mice repositories

‣ download the BIND 9 RPM from []<arch>/<version>/

root-shell> yum install ISCBIND-<version>-<flavor>RHL<arch>.rpm 
root-shell> mkdir /var/named 
root-shell> useradd -d /var/named -r named 
root-shell> chown -R named: /var/named 

‣ create a BIND 9 configuration file '/etc/named.conf'

options { directory "/var/named"; dnssec-validation auto; }; 

‣ create an 'rndc' configuration

root-shell> rndc-confgen -a 

‣ verify the configuration

root-shell> named-checkconf -z 

A basic configuration file for BIND 9 "named"

Below is my basic /etc/supervisord.conf configuration file for one service, the BIND 9 DNS Server:

file = /tmp/supervisor.sock 
chmod = 0777 
chown= nobody:nobody 

supervisor.rpcinterface_factory = supervisor.

logfile = /var/log/supervisord.log 
logfile_maxbytes = 10MB 
loglevel = info 
pidfile = /var/run/ 
identifier = supervisor 
directory = /tmp 

command=/usr/sbin/named -u named -f 

Starting supervisord

With the configuration file in place, we can start supervisord. Make sure that BIND 9 is not started or you will end up with two instances of the BIND 9 server running, which isn't recommended. Also make sure that supervisord will be started on reboot of the server, either through a startscript or other means. Note that the supervisord packages bundled with Linux distributions install a startscript.

root-shell> /etc/init.d/supervisord start 
root-shell> rndc status 
version: 9.9.6-P1 <id:3612d8fb> 
number of zones: 98 
debug level: 0 
xfers running: 0 
xfers deferred: 0 
soa queries in progress: 0 
query logging is OFF 
recursive clients: 0/0/1000 
tcp clients: 0/100 

server is up and running 
root-shell> ps -ef 
root 10906 0.0 2.5 209096 12988 ? Ss 19:55 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf 
named 10908 0.7 1.6 44292 8112 ? S 19:55 0:00 /usr/sbin/named -u named -f 
root 10910 0.0 0.2 110228 1156 pts/0 R+ 19:55 0:00 ps aux 

root-shell> supervisorctl status 
named RUNNING pid 10908, uptime 0:03:19 
root-shell> chkconfig --add supervisord 
root-shell> chkconfig supervisord on 

Great, supervisord has started, and it also started the BIND 9 process "named". DNS is working now.

Simulating a BIND 9 crash

To simulate a BIND 9 crash, we "kill" the BIND 9 named process:

root-shell> killall -9 named 

Supervisord should detect that the running BIND 9 process has terminated, and start a new one. DNS is still up and running.

Controlling supervisord

Supervisord can be controlled from the command line using the supervisorctl command. A list of all a control commands can be found with "help", and a description of each command with "help command":

shell> supervisorctl help 
default commands (type help ): 
add clear fg open quit remove restart start stop update 
avail exit maintail pid reload reread shutdown status tail version 

shell> supervisorctl help status 
status Get all process status info. 
status Get status on a single process by name. 
status Get status on multiple named processes. 

shell> supervisorctl status named 
RUNNING pid 25770, uptime 0:00:12 

shell> supervisorctl stop named 
named: stopped 

shell> supervisorctl start named 
named: started 

Now, whenever there is a triggered assertion error in the code BIND 9 will terminate, but supervisord will bring it back from the dead. Your DNS service stays up, and your users and customers stay happy.

Read the supervisord documentation on how to setup event notifications, so that you get an e-mail notification should BIND 9 restart (should the outage be caused by a security vulnerability you might want to report it to as well).

Of course supervisord can be used to restart other processes as well, including other types of DNS Servers (NSD, Unbound, dnsmasq ...).

Men & Mice 2014 Holiday Greeting

Men & Mice Christmas greeting 2014

Men & Mice thanks you for your business this year. It has been a pleasure helping you reach your goals, and we look forward to contributing to your success in 2015. We wish you a prosperous and happy new year! 

Men & Mice staff


Unparalleled support for DNS Servers and tightened Security


Men & Mice announces the release of version 6.7 of the Men & Mice Suite.

The Men & Mice Suite is the ideal tool for network managers who need superfast daily management, planning, reporting and auditing on growing dynamic IP networks, delivering the added benefit of improved network security as well. 

Unparalleled support for DNS Servers

To ensure the solution will scale with businesses as they grow, the Men & Mice Suite integrates with the widest available range of DNS servers, such as BIND, Microsoft DNS services and Unbound. The 6.7 edition adds PowerDNS to enable customers to run hybrid environments for tightened securityIn this release Men & Mice takes flexibility one step further with the addition of Amazon Route53 DNS services support.  Enterprises moving to the AWS cloud or running hybrid private/public clouds can now keep full control of their DNS, DHCP and IP environment with the Men & Mice Suite.

Support for Amazon Route53
The Men & Mice Suite now supports Route53, Amazon’s cloud DNS service. With this integration, users can manage DNS information stored on the Amazon Route53 DNS servers in the same way they can manage DNS on other supported platforms, such as creating new zones and edit DNS records in existing zones.

Support for PowerDNS
PowerDNS, an open source, high performance DNS server, is now supported in the Men & Mice Suite.  This capability will especially benefit customers with complex hybrid environments, as they will be able to  manage all their diverse DNS servers from one solution, regardless if they are BIND, Microsoft DNS or PowerDNS servers. 

DNS Security

The increase of mobile devices (BYOD), the Internet of Things (IoT) and the growth of cloud-based virtual machines has caused a seismic shift in the DDI landscape, leading to greater awareness of network-related security risks. Security manifests itself in various formats, such as availability, performance and the ability to withstand attacks like DDoS Attacks, DNS cache poisoning and other DNS security threats. The Men & Mice Suite helps network administrators address such risks by offering hybrid DNS server support and high availability.

The 6.7 edition of the Men & Mice Suite adds DNS and DHCP service Monitoring and support for TLSA records that enable the storage of/and signing keys that are used to verify SSL/TLS certificates through DNSSEC.

DNS and DHCP service Monitoring

The Men & Mice Suite now actively monitors the status of the DNS and DHCP services on all managed platforms and will alert users if the services become unavailable.  In addition to being displayed in the user interface, the alerts can be sent to monitoring systems for further processing.  This will serve to maximize availability and enable customers to avoid costly unscheduled downtime.

Support for TLSA records
TLSA records,  in conjunction with DNSSEC signatures, provide an easier and more secure way for applications such as Web browsers and mail servers to authenticate SSL/TLS certificates.   Support for management of TLSA records has been added to the Men & Mice Suite.  For more info on TLSA and DANE (DNS-based Authentication of Named Entities), users can view a recent Men & Mice webinar on the topic.

Reverse zone improvements
Handling of reverse records and reverse zones has been enhanced in this new version and is now much more tightly integrated into the IPAM module.  Users can select any number of subnets and create and/or update the corresponding reverse entries for the subnets.  Reverse record (PTR records) details are now also included with the IP address details in the IPAM view.


Role-based access support

Role-based access allows customers to create roles in the Men & Mice Suite and assign these roles to users and groups.  All supported users and groups, whether Men & Mice built-in or from Active Directory or Radius can have roles assigned to them, which will greatly simplify access administration while providing a more flexible access model.  


Men &amp;amp; Mice Suite version 6.7FREE TRIAL


or Call us at +1 408.516.9582 to speak to a sales representative.

New features in version 6.7

DNSSEC & DANE – E-Mail security reloaded


Conventional TLS/SSL for SMTP is flawed and the x509 certificate business is a pain

Get a 35 minute crash course in "DANE" style securing x509 certificates using DNSSEC secured DNS using BIND 9 and the Postfix mail server from Mr. Carsten Strotmann from the Men & Mice Services team.

Carsten also answers the following questions:

  • Does the Men & Mice Suite support DANE & TLSA?
  • For DANE I need a DNS hosting provider that supports DNSSEC, where can I find one?
  • What impact will DANE have on the current certificate business?

Take a look!


Tags: , , , ,

Introduction to the Men & Mice Web Service API


The Men & Mice Suite provides a large number of powerful features to manage DNS, DHCP and IP infrastructures of all sizes. All of these features are accessible through the Men & Mice GUI client or the web interface that come with the Suite.

An alternative to using the graphical tools, is to use the Men & Mice web service API to automate repetitive tasks. The web service has even been used to build new custom applications on top of the Men & Mice Suite. In this article, I will start by giving a brief overview of the API, followed by a short example to demonstrate how the service can be used.


The Men & Mice web service API first came out in 2008, and has been in constant development ever since. It uses the Simple Object Access protocol (SOAP) and has a Web Service Description Language (WSDL) definition that describes all operations that can be done. All new features in the Suite are implemented for the web service, so everything that can be done using the GUI tools can also be done through the API. The latest official API documentation can be found here. You can also see what operations are supported by your current release by connecting to the Men & Mice Central server that comes with the Suite (see the official API documentation for further details).

Examples of what users are doing with the Men & Mice web service include:
  • Search for DNS records in all zones that match a particular name.
  • Change the time to live (TTL) for all DNS records containing a certain pattern.
  • Construct a report for DHCP scope options and address pools, and e-mail to responsible personnel.
  • Automatically create DHCP reservations when deploying virtual machines.
  • Create DNS zones in bulk according to a template.
  • Manage a list of blacklisted DNS zones.
  • Run periodically a cleanup of IP addresses that haven't been seen on the network for some fixed time period.
  • Find the next free IP address, and create records and/or reservations through own web portals.

Numerous SOAP clients exist for different programming languages. For this article, we will be using the python programming language and the suds SOAP framework.

The first thing to do, when using the web service is to download the WSDL file from the Men & Mice Central server, that defines what SOAP commands are available for your release. The next step is to log into the system using the Login command to retrieve a session token that will be used to authenticate all subsequent calls to the service. Since the session token has to be used for all requests to the service, it becomes very repetitive to manually include it in all calls. We will, therefore, use a wrapper client, available from the Men & Mice website that will automatically append the token to all subsequent calls, once we have logged onto the system. Suds also requires a slight modification of the SOAP envelope in order to work with the Men & Mice web service, which will be handled by the client.

import suds
from soapCLI import mmSoap
    cli = mmSoap(proxy="",server="",
             username="administrator", password="secret")
except suds.WebFault as e:
    print "Error while logging into the Men & Mice suite: %s" % e.fault.faultstring

The same authentication model applies to users logged in through the web service, as to users using the GUI. The user needs to have permissions to work with a given resource (DNS records, zones, servers, users, etc). Special permissions are also needed to execute commands through the web server interface. The permissions can be granted by an administrator using a GUI, or the web service.

Error handling

The SOAP specification defines a format for messages containing error information and is used by the Men & Mice web service to notify a client when an operation fails. In some cases, we might also get partial failures. If we are for example deleting DNS records from a number of servers, then some of the DNS servers might be unreachable, while other servers successfully remove their corresponding records. In that case the operation will return a collection of error messages describing the errors.

Working with resources

All objects within the Men & Mice Suite, whether they are IP addresses, DNS zones, DNS records, DHCP scopes or any other type of resource, have a unique resource identifier that can be used to reference, retrieve and work with the resource. It is also possible to fetch a collection of items, for example DNS records and search the retrieved items for a specific pattern. Many of the SOAP commands also provide a filter input parameter, in order to filter what records to return from the service. Further details about references and filters can be found in the official Men & Mice API documentation .

Example usage

We will now demonstrate a simple, yet realistic example of how the API might be used.

The example demonstrates a scenario where we need to relocate a large number of websites hosted on different machines to a new web server. Instead of changing each DNS record manually by hand, we will automate the task by using a simple script.

We start by logging into the system using the before mentioned client available from the Men & Mice website. We then create an array of DNS records (arrayOfDNSRecordsToAdd) that will be used to store the new records that we want to add. We also create an array to store unique references to the records that we want to delete (dnsRecordsToRemove), after we have saved the newly created records.

    cli = mmSoap(proxy=proxy,server=server,
             username=username, password=password)
except suds.WebFault as e:
    print "Error while logging into the Men & Mice suite: %s" % e.fault.faultstring
    return False
arrayOfDNSRecordsToAdd = cli.create("ArrayOfDNSRecord")
dnsRecordsToRemove = cli.create("ArrayOfObjRef")

In order to get all DNS records, we need to fetch all DNS zones in the system using the GetDNSZones command. Once we have an object identifier for each zone (dnsZoneRef), we can use the GetDNSRecords SOAP command to retrieve all DNS records that are in the zone. Two filters are used. The first filter is used to make sure that only master zones are retrieved. The second filter is used to limit the results to records of type "A". If a record points to any of the servers we are migrating from (IP addresses, or, then we add the record to the collection of records we want to remove and create a new DNS record pointing to the web server we are migrating to (IP address

# Retrieve all master zones
(_, arrayOfDNSZone), (_, totalResults)  = cli.GetDNSZones(filter="type:Master")
if totalResults == 0:
    print "No DNS zones found."
    return True
for dnsZone in arrayOfDNSZone.dnsZone:
    # Fetch all A records from the zone 
    (_, arrayOfDNSRecord), (_, totalResults) = cli.GetDNSRecords(dnsZoneRef=dnsZone.ref, filter="type:^A$")
    if totalResults > 0:
        for dnsRecord in arrayOfDNSRecord.dnsRecord:
            if in ["", "", ""]:
                # Modify records to point to new server
                dnsRecord.ref = None
       = ""

Once we have created all the records, we add them to the system using the AddDNSRecords command. The command returns an array of record references for each new DNS record that has been added, along with an array of errors, if any. The references are used as a parameter to the GetDNSRecord command, that we use to get further information about the records.

The retrieved arrayOfDNSRecordRef will have the same record order as the arrayOfDNSRecordsToAdd that was passed to the SOAP command. If an error occurred when creating a particular record, then a "null" reference "{#0-#0}" will be returned for that record instead. The results could be used to make sure that any old records that correspond to the ones we were unable to add, will not be removed in the next step when we delete old records. We will, however, omit that step for this example and instead prompt the user to do a manual cleanup in case of any errors.

if len(arrayOfDNSRecordsToAdd.dnsRecord) > 0:
    # Create new records
    (_, arrayOfDNSRecordRef), (_, addRecordArrayOfErrors) = cli.AddDNSRecords(dnsRecords=arrayOfDNSRecordsToAdd, 
                                                                                saveComment='Modifying DNS records to point to host "".')
    if len(arrayOfDNSRecordRef.ref) > 0:
        for recordRef in arrayOfDNSRecordRef.ref:
            if recordRef == "{#0-#0}":
                # Caused by a record that was not added due to some error
                print "Added record:", cli.GetDNSRecord(dnsRecordRef=recordRef)
            except suds.WebFault as e:
                print 'Unable to retrieve record with ref "%s" due to the following error: %s' % (recordRef, e.fault.faultstring)

If any errors came up while adding new records, then we instruct the user to do a manual clean up. Note that the error property from the AddDNSRecords response is "nillable". That means that it might not be present in the response. We therefore check if it has been set, before using it.

If no errors came up while adding new records, then we remove the old records using the RemoveObjects operation. The RemoveObjects command can be used to delete almost any object in the system, given that we have a resource reference to the object.


if hasattr(addRecordArrayOfErrors, "error") and len(addRecordArrayOfErrors.error) > 0:
    print """One or more errors occurred while adding records: %s.
 Old records will not be deleted. Please check manually and delete old records that were successfully migrated.""" % addRecordArrayOfErrors.error
    return False
    # No errors came up during migration, we delete the old records
    removeRecordErrors = cli.RemoveObjects(objRefs=dnsRecordsToRemove,
                                            saveComment="Removing records that now point to host %s." % addressTo)
    if len(removeRecordErrors) > 0:
       print "The following errors occurred while removing old records:", removeRecordErrors
       return False

The complete example can be downloaded here.


The Men & Mice web service API can be used to automate repetitive DNS, DHCP and IP address management tasks and can be used to write custom applications on top of the Men & Mice Suite. The service can be used by most common programming languages, and as demonstrated, is easy to use.

[Webinar] IETF 90 Report – DNS, DHCP, IPv6 and DANE


Report from IETF 90 in Toronto 2014

The IETF, Internet Engineering Task Force, those that are working on new Internet Standards, met in Toronto in July 2014.IETF resized 600

Mr. Carsten Strotmann from the Men & Mice Services team gives an overview of interesting developments from the working groups inside the IETF, after attending online at the IETF 90 in Toronto.

Hear more on:

  • DNS
  • DNS-Privacy
  • IPv6
  • DANE
  • DHCP(v6)
  • and new RFCs that have been published since the last IETF in March 2014

Fetch the slides, webinar recording and link collection.

Tags: , , ,

RIPE 68 report


Report from RIPE 68 in Warsaw, Poland

A RIPE Meeting is a five-day event where Internet Service Providers (ISPs), network operators and other interested parties from all over the world gather.

In this webinar, Carsten Strotmann from the Men & Mice Services team reports about what was new at the RIPE 68 meeting.

Hear what he had to say on:

  • Amplification DDoS Attacks – Defenses for Vulnerable Protocols
  • news from DNS-OARC meeting (DNS measurements, open resolver stats)
  • Strengthening the Internet Against Pervasive Monitoring
  • What Went Wrong With IPv6?
  • RIPE IPv6 Analyser
  • IPv6 troubleshooting procedures for helpdesks
  • Using DDoS to Trace the Source of a DDoS Attack
  • Measuring DNSSEC from the End User Perspective
  • Google DNS Hijacking in Turkey
  • The Rise and Fall of BIND 10
  • Knot DNS Update – DNSSEC and beyond
  • Bundy-DNS – the new life of BIND 10

Have a look at the slides and recording from the webinar to learn more.


Men & Mice Suite version 6.6 released


Men & Mice, a leading provider of DDI solutions, announces the release of the Men & Mice Suite version 6.6 along with added functionality for the Men & Mice Appliances.

The new release focuses on usability enhancements and DNS Security features, further ensuring that the Men & Mice Suite retains its position as one of the most reliable and user - friendly solutions available.

Highlights in version 6.6:

Utilization of static subnets displayed in the Management Console and the Web UI
Real-time utilization of static subnets is now displayed in the user interfaces, which allows users and administrators to quickly see the utilization percentage of the subnets.  The utilization information can be copied out from the console for easy reports and the users can furthermore sort and filter by the utilization.  Filtering of the utilization can be combined with other filters so the user can, as an example, get a list of all subnets of a specific size, or of a specific type that are more than 85% utilized.

With the addition of Smart-filters one of the most popular features of the Men & Mice Suite has now become even more powerful.  The user can save filters and place them in "smart" folders. The user can also right-click the filter, change its name and the filter statement.  Filters created by the "administrator" user are global, i.e. they are visible by all users.

Support for RPZ (DNS Firewall)
Men & Mice now support the Response Policy Zone framework in BIND, which is the underlying mechanism of DNS Firewalls.  Administrators can create and define RPZ zones with the Men & Mice Suite or, via the tool, configure the DNS servers to subscribe to RPZ feeds from trusted sources.

SNMP Profiles and SNMPv3
Multiple SNMP profiles can now be created.  This allows enterprises with complex networks that use the Men & Mice Suite to pull discovery information from routers that belong to different security realms to create a profile for each realm.  With the addition of SNMPv3, profiles can be for version 1, 2c or 3 and can contain different settings, such as authentication and community strings.

Appliance diagnostic access
The Men & Mice Appliances, both DNS/DHCP and Caching, now have a read-only diagnostic shell access that can be used to run troubleshooting commands (such as dig, drill, etc.) and to gather information and logs from the appliances.  The access is read-only so no changes can be made to the configuration or data on the appliances using the diagnostic access.

Backup and restore of appliances
The Men & Mice Central now stores a backup of full configuration and data from all the managed appliances.  Full backups are taken daily and incremental backups are taken each time a change is made on the appliances.  If an appliance were to become unavailable for some reason, a new appliance can be configured with the same IP address as that appliance. All configurations, including DNS and DHCP data, network configuration and other settings is restored from the backup to the new appliance making it identical to the previous one.



Men &amp;amp; Mice Suite version 6.6FREE TRIAL


On-link vs Off-Link Packet Delivery: Unicast, Multicast, Anycast


By David Beck, a Men & Mice trainer and course developer

Delivering IP packets within a subnet (on-link) is different than delivering between subnets (off-link). This article examines those differences for unicast, multicast, and anycast destination addresses. The goal is to shine a light on the differences for anycasts and to explain on-link anycasting. Everything is applicable to both IPv4 and IPv6, with one exception that is clearly noted. However, that difference is why this is fundamentally an IPv6 article. The IPv6 terminology "link" will be used in lieu of "subnet." "Node" will be used in lieu of "host."


A unicast address uniquely identifies one interface on one node. It is used to deliver a packet to only one interface. It is the type of address that initially comes to mind, when one hears "IP address." A sender with a unicast destined packet looks up the packet's destination address in its routing table, and selects the longest-match.

The longest-match entry may indicate that the destination is on-link (OL). For an OL destination the sender uses the data link layer (DLL) protocol, e.g. Ethernet, to deliver the packet to the final destination. Alternatively, the longest-match may indicate that the destination is off-link (XL, external-link). The routing table entry for an XL destination includes an OL router that is closer to the destination. The sender uses the DLL protocol to deliver the packet to the router. The router repeats the same process sending the packet toward the final destination.

An internet is two or more links (networks) joined by a router. It is the need to deliver unicast packets to XL nodes, that spurred the creation of internet protocols such as IP. If all destinations were OL, packet delivery could be done with only the DLL protocol and DLL addresses. (Note that the Internet is a huge internet with links throughout the world joined by routers.)


A multicast address identifies multiple interfaces on multiple nodes. A packet is delivered to all identified interfaces. In both IPv4 and IPv6, multicast addresses are distinct from unicasts and easily identifiable. IPv4 multicasts are (addresses starting with 224-239). IPv6 multicasts are ff00::/8. Multicasting was not always a part of IP. It was added at the end of the 1980s.

Delivery of OL IP multicasts relies on the capability of the underlying DLL protocol to support multicasting, or to at least support broadcasting. Happily, Ethernet and 802.11 wireless have multicasting functionality and there are multicast MAC addresses. This makes OL multicasting delivery simple. IP multicast addresses are mapped to MAC multicast addresses through a formula, for IPv4 that is defined in RFC "Host Extensions for IP Multicasting" (, and for IPv6 in the RFC “Transmission of IPv6 Packets over Ethernet Networks” ( A node assigned a multicast IP address configures its network interface to listen for the corresponding MAC multicast address. When a packet is sent to an OL IP multicast address, it is packed in a frame destined for the corresponding MAC multicast. All nodes listening for the MAC address, receive and process the frame and the packet. OL multicasting is easily implemented, and widely used.

Both IPv4 and IPv6 routing protocols rely on OL multicasting. A router sends one packet to notify all others of routing protocol information, such as a network topology change. (OL delivery of IP multicasts, with underlying DLL protocols that don't support multicasting or broadcasting, can be very challenging. That little nightmare is happily well outside the scope of this article.)

XL multicasting, with a multicast address assigned to nodes on different links, is far more challenging. A special multicasting routing protocol must be implemented. The sender generates one packet; multicasting routers must duplicate the packet to deliver it to nodes on different links. XL multicasting is not widely used. The mass majority of organizations doesn't implement multicasting routing protocols. Multicasting routing protocols are not used in the open Internet.


Now we're reaching the heart of this article.

Like a multicast, an anycast is an address assigned multiple interfaces on multiple nodes. Unlike a multicast, an anycast packet is delivered to only one node. The sender doesn't care which node receives the packet, all the destinations are equivalent.

Like multicasting, there is a wide chasm between OL and XL anycasting. For multicasting, OL is easy and widely used. XL multicasting is challenging and rare. It is the opposite for anycasting. XL anycasting is easy.

XL anycasts are used for root and TLD (Top Level Domain) DNS servers. For example, the F root server ( has the IPv4 address The address is an anycast. It is currently assigned to fifty-five DNS servers throughout the world ( For DNS, all these servers are identical. When a packet is sent to, the natural process of unicast routing delivers it to only one node (one DNS server). Since all the servers have the same information, it is unimportant which is reached. In order for administrators to manage individual servers, each also has a unicast address, but that address is not intended for answering DNS queries.

XL anycast addresses are indistinguishable from unicast addresses. When a unicast address is assigned to a second node, the address becomes an anycast. The various nodes assigned the address don't know that it is an anycast and not a unicast. No special handling is required by nodes assigned an anycast address. No special handling is required by a sender generating a packet to an anycast destination. There is no protocol supporting anycasts. None is needed. Normal unicast routing handles delivery of XL anycasts. The only requirements are that the nodes sharing an address are on different links, and routing is properly configured. There isn't even an RFC defining the technical specifics for XL anycasting. Note that several RFCs discuss anycasting. For example the RFC "Operation of Anycast Services" is a Best Current Practices (BCP) document (

While XL anycasting is found in IPv4 and IPv6, OL anycasting is only implemented in IPv6. Nodes on the same link share an address. A packet addressed to an OL anycast is still delivered to only one interface. However, because the nodes are on the same link, routing can't handle delivery. Unlike XL anycasting, OL anycasting requires a technical specification. RFC "IP Version 6 Addressing Architecture" ( specifies OL anycasting.

A node assigned an OL anycast sends Neighbor Advertisements (NAs). The NAs associate the anycast address with the node's own DLL unicast address. The NAs are fundamentally the same as the node would send for an assigned unicast address. Mapping an OL anycast, to the DLL unicast of a node, was a fundamental implementation decision taken by the IPv6 designers. Without prejudice as to whether it was a good or bad decision, it is noteworthy that there were other options. For example, First Hop Redundancy Protocols (FHRPs), essentially implement an OL anycast address, but with a completely different approach. The IETF standardized FHRP is RFC specified: "Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6" (

By definition, an anycast address is assigned to multiple nodes. Each node assigned an OL anycast sends NAs associating the address with its own DLL unicast. These NAs are basically competing with each other. Other nodes on the link associate the OL anycast with one DLL unicast address, with one of the nodes assigned the OL anycast. If, three nodes on an Ethernet share an anycast address, another OL node will use the unicast MAC address of one of the three. When a packet is sent to the anycast address, the DLL deliveries it to one node. In this way, the anycast requirement of delivery to a single node, is met.

It is worth considering why DLL unicasting is used for OL anycast delivery. While it isn't ideal, it is necessary because DLL protocols don't have anycast capabilities. If, for example, Ethernet had anycasting addresses and anycast functionality, an OL anycast could be mapped to a MAC anycast. OL anycasting would then be as trivial as OL unicasting, and OL multicasting. Since DLL anycasting doesn't exist, OL anycasts must be mapped to what DLLs provide, unicasts, broadcasts, or multicasts. DLLs broadcasting and multicasting functionality cannot be used, because they violate the anycast requirement of deliver to only one node. So delivery of an OL anycast is necessarily implemented by a DLL unicast.

XL anycast addressing require no special handling. OL anycast addressing does. A node assigned an OL anycast must be explicitly informed that the address is an anycast, requiring IPv6 systems to implement a technique to indicate anycasts. The OL anycast indication suppresses Duplicate Address Detection (DAD). When an IPv6 unicast is assigned, DAD tests if the address is assigned to another node on the link. The address isn't used if DAD reports duplication. Anycasts are purposely assigned to multiple nodes, so DAD is disabled for OL anycasts. Additionally, OL anycast NAs are sent with a slight delay, and, more importantly, with the Override flag cleared in the NA message.

A set Override flag tells the receiver to replace a previously cached entry for the advertised IPv6 address. For a unicast NA, the Override flag is set, so that a receiver uses the new advertisement. This makes sense since the unicast NA is coming from the same node that sent the older cached information. Newer unicast information is better. For an OL anycast, each node assigned the address sends an NA with its own DLL address. The cleared Override flag in these NAs means a receiver will use the DLL address from the first NA that arrives. Older anycast information is better.

The cleared Override flag prevents oscillating between different DLL addresses for the anycast destination. It additionally means that, at any given time, senders on a link are reaching different nodes that share the anycast address. This can be viewed positively, as it provides load balancing. However, it can't be controlled and all senders could also be sending to one DLL address (to one node). Unfortunately, if a cached OL anycast destination becomes unreachable, it can't be replaced until the cached entry times out. So for the unlucky nodes on the link with the unreachable address cached, the OL anycast is unreachable, but for all other nodes on the link the anycast is reachable.

So unlike an XL anycast, an OL anycast requires both special configuration and special handling on the nodes assigned the address. Neither OL or XL anycasts require special handling by senders.

RFCs define IPv6 OL anycast addresses. Being defined distinguishes them from other anycasts, which are purposely indistinguishable from unicasts. The most common OL anycast is the Subnet-Router anycast (SRA). Every link has an SRA address. All the interface identifier bits set to 0, identifies the SRA. For the link 2001:db8:cafe:fee::/64, the SRA is 2001:db8:cafe:fee::/128. The SRA is defined in RFC 4291. RFC "Reserved IPv6 Subnet Anycast Addresses" ( reserves the highest 128 addresses on each link for OL anycasts.

Initially it may seem that the SRA would be an ideal way to implement a FHRP. However, the widely used FHRPs, HSRP, VRRP, GLBP and CARP all support IPv6 without using the SRA. Where is the SRA used? Saying it has not been widely implemented, is overstating its current uses. Depending on feedback, perhaps another article will delve deeper into the SRA.

So it ends on a whimper. IPv6 has added OL anycasting, but it is unused by any significant application, and its usefulness is questionable.


IP unicasting, both OL (on-link) and XL (external-link, off-link), is easy and widely implemented.
OL IP multicasting is easy and common. XL multicasting is more involved and uncommon.
OL IP anycasting is extremely rarely used and limited to IPv6. XL anycasting is easy, and although not common, widely used for DNS.

Tags: ,

Immediate ROI with the Men & Mice Suite during transition to Open Source DHCP


Texas Woman's University (TWU) is a major multi-campus U.S. public university, primarily for women. Texas Woman's UniversityIts campuses in Denton, Dallas and Houston are joined by an e-learning campus offering innovative online degree programs in business, education and general studies. 


This University needed better control and increased flexibility for a variety of network administration tasks, with the immediate need being a smooth transition from Windows to Open Source DHCP.

"Managing the Transition to Open Source DHCP “A Major Selling Point.”

“I am a proponent of open source technology,” said the College’s Lead Network Administrator, “and converting to Linux had been on my list of goals for a long while. I’d built a test Linux DHCP server, but I ran into some difficulties modeling the database migration.” So he did what any network administrator might do: he went looking for help online. “Basically I was trying to convert things cleanly and completely from Windows to Linux, and I was looking for a tool that would help me do that. I posted my needs on message boards and got a recommendation from another network administrator: “Try Men & Mice”. Not only did it help him make a smooth transition to Linux, but he adds, “I was able to get much better control over my Windows DHCP server right away. We are now running all of the DNS servers through the Men & Mice management console, as well. I am very happy with it.”

While helping the University accomplish their transition to Open Source DHCP, Men & Mice also expedited their ROI by also helping them to get a handle on their IP address management which was still being handled on an excel spreadsheet. Men & Mice Really Simplifies the IPAM Management Piece.”

In addition, the robust API that Men & Mice includes was used to mitigate DNS coding errors and security concerns by automating DNS procedures previously not possible in the home grown application they were using. To accelerate the shortened ROI,  the University also used the tool to help clean up stale PTR records in a minimal amount of time which was “a huge benefit and time savings”.

The Results:

“A Significant Benefit for us.”

From a network administrator’s perspective, success can be measured in a number of ways, and for the University Office of Technology, one of the most meaningful measures is user satisfaction. “We are here to facilitate people’s education. Our students are here to better their lives, and we are here to support them. It’s an important mission, and when technology problems interfere with that, people will let you know quickly. Since we installed the Men & Mice Suite, I haven’t heard a thing.”

Read the full case study on how TWU, with the Men & Mice Suite, put their focus on facilitating people’s education instead of mundane network management and troubleshooting tasks.

All Posts