The Men & Mice Blog

An Introduction to the Men & Mice Web Service: JSON-RPC

Posted by Men & Mice on 2/23/15 10:48 AM

The Men & Mice Suite is a comprehensive DDI solution that allows you to manage your DNS, DHCP and IP Addresses in a simplified, flexible manner. In order to maximize this flexibility the Men & Mice Suite provides a rich set of commands accessible through its web service interface. These commands are used internally for almost all requests between the user interface and the Central service, so if something can be done in the user interface it can also be done through the web service interface.

Initially the web service interface provided a SOAP API and a WSDL service. Starting with version 6.6, the Men & Mice Web Service also provides a JSON-RPC service that complies with the JSON-RPC 2.0 specifications. The main reason for this decision to provide JSON-RPC along with SOAP is that JSON is well supported in popular programming languages such as JavaScript and Python, and working with JSON-like data instead of XML is much easier and less error prone, not to mention that JSON packets are smaller than XML packets and more human readable.

If you are interested in the SOAP service there is an excellent introduction to it on our blog (see: Introduction to the Men & Mice Web Service API).

JSON-RPC

The JSON-RPC service supports all the same commands as the SOAP API. You can find a complete list of the SOAP commands available here. The same code is used internally in the Men & Mice Suite so when a new SOAP command is added, it will be readily available as a JSON-RPC method. As with SOAP, our main focus is to provide JSON-RPC over HTTP/HTTPS.

No special magic is needed to send a JSON-RPC request. When our web service receives a packet it will determine from the content of the packet whether this is a SOAP or a JSON-RPC request. You can even mix those two and the web service will simply respond with a JSON if this was a JSON-RPC request or an XML if this was a SOAP request. The general format of a Men & Mice Web Service JSON-RPC request and a response is as follows:

--> {"jsonrpc": "2.0", "method": "theMethodName", "params": theParams, "id": 1}
<-- {"jsonrpc": "2.0", "result": theResult, "id": 1} 

The jsonrpc member should always be "2.0". The method member should contain a string with the name of the method to be executed. The params member contains data that should be passed to the method. The member id is optional, but if it exists the response will contain the same ID as provided in the request. If successful the response will include a member result that will contain the result from the RPC method that was executed. If an error occurs the response will not include a result but instead another member named error that will include information about what happened.

An example:

--> {"jsonrpc": "2.0", "method": "Login", "params": {"password": "secretPassword", "loginName": "administrator", "server": "10.5.0.6"}, "id": 1}
<-- {"jsonrpc": "2.0", "result": {"session":"J95A5uh9obM6KjCtbtHZ"}, "id": 1}
    
<-- {"jsonrpc": "2.0", "method": "GetDNSServers", "params": {"session": "J95A5uh9obM6KjCtbtHZ"}, "id": 2}
--> {"jsonrpc": "2.0", "result": {"dnsServers":[{"ref":"{#2-#10}","name":"caching1.demo.","resolvedAddress":"10.5.0.27","port":1337,"type":"Unbound","state":"OK","customProperties":[],"subtype":"Unbound"},{"ref":"{#2-#11}","name":"caching2.demo.","resolvedAddress":"10.5.0.30","port":1337,"type":"Unbound","state":"OK","customProperties":[],"subtype":"Unbound"}],"totalResults":2}, "id": 2} 

JSON-RPC and Python

Python is powerful as a scripting language. The standard libraries are pretty comprehensive and if you need something more specific then there are plenty of additional libraries that can be found. One of the strengths of Python is its dynamic type system and "duck typing" that allows for the creation of complex objects without writing too much code.

We have created a small, light-weight Python module that uses the request library to simplify session handling and data conversion from Python dictionaries to JSON requests. The class overloads the __getattr__ method, so that all unknown functions are assumed to be RPC requests and all the arguments passed are assumed to be a RPC data.

You can find the latest version of the Python module here. To install it, extract the downloaded file, go into the subdirectory and run:

$ python setup.py install

Before we can use any of the commands we need to create a JSONClient object and log in to our Central Service. Let's start Python and type in the following commands.

$ python
>>> import mmJSONClient
>>> client = mmJSONClient.JSONClient()

The Basics

The Login method accepts the following arguments: proxy, server, username and password. Proxy is the name or IP address of a server running the Men & Mice Web Service. If you do not provide the proxy argument, then the Login method assumes the web service is running on the same machine as the Central Service. The server argument should contain the name or address of the machine that is running the Men & Mice Central Service. The username argument is the name of the user that we want to log in as and password is his password. Note that in order for this user to be able to use the web service he has to have permissions to use the web user interface. For optimal security, it is best to create a new dedicated user account that only has access to the objects the script needs to function.

Now let's log in to the server. In this example I'm logging in to a server named 'central.demo' with the user name 'a_user' and the password 'secret'. Since the web service is also running on 'central.demo' I don't need to provide the proxy argument.

>>> client.Login(server='central.demo', username='a_user', password='secret') 

As I mentioned earlier, we can use all the commands that are available for the SOAP API, and there are a lot (around 200 at last count). A list can be found here, along with descriptions of what arguments they need and what data they return.

Let's start with something simple, like the GetDNSServers method. As the name implies, the method will return all the DNS servers available to the user. Note that it might not return all the DNS servers available, but instead only those the user has permission to see. If you look at the description of GetDNSServers, you will see the only argument required is the session ID; everything else is optional. The session ID is a random string that the web service returned from the Login method. If you have successfully executed the Login method, you can see what your temporary session ID is by doing:

>>> print(client.sessionid)
VFr79UQMhFfOq4Q8l0Qi 

Or if you are using Python 2.7, use print without parentheses:

>>> print client.sessionid
VFr79UQMhFfOq4Q8l0Qi 

The rest of this article assumes use of Python 3.x. When using earlier versions of Python the syntax may have to be adjusted accordingly.

Fortunately when using the JSONClient class you don't have to worry about the session ID, as the class handles that automatically so if you want to list all the DNS servers available, you can simply type:

>>> print(client.GetDNSServers())
{u'totalResults': 2, u'dnsServers': [{u'name': u'caching1.demo.', u'resolvedAddress': u'10.5.0.27', u'ref': u'{#2-#10}', u'subtype': u'Unbound', u'state': u'OK', u'customProperties': [], u'type': u'Unbound', u'port': 1337}, {u'name': u'caching2.demo.', u'resolvedAddress': u'10.5.0.30', u'ref': u'{#2-#11}', u'subtype': u'Unbound', u'state': u'OK', u'customProperties': [], u'type': u'Unbound', u'port': 1337}]} 

The result from GetDNSServers is a Python dictionary that contains two members: totalResults and dnsServers. As stated in the SOAP API documentation, the totalResults member contains the number of DNS Servers we have access to, and the member dnsServers contains an array of DNS servers. To refer to a member in a Python dictionary we can use brackets - for example, the following can be used to print out the total number of DNS servers:

>>> print(client.GetDNSServers()['totalResults'])
2 

Providing arguments to a method is simple. GetDNSServers has a number of optional arguments, for example the filter argument. Filter is powerful argument that is provided with many of the Men & Mice SOAP API commands. It allows you to limit the result to only the items that you want to see. You can use wildcards and regular expressions with a filter. For more information, look at the description of filtering in the SOAP API. Let's say that we wanted to get all the servers that contain the number 2 in their name. We can simply do:

>>> print(client.GetDNSServers(filter='name:2'))
{u'totalResults': 1, u'dnsServers': [{u'name': u'caching2.demo.', u'resolvedAddress': u'10.5.0.30', u'ref': u'{#2-#11}', u'subtype': u'Unbound', u'state': u'OK', u'customProperties': [], u'type': u'Unbound', u'port': 1337}]} 

Or let's say we wanted to print out all A records from the zone 'applepie.ak.is' that start with the name 'apple':

>>> print( '\n'.join(['{}\t{}\t{}'.format(r['name'],r['type'],r['data']) for r in client.GetDNSRecords(dnsZoneRef= 'applepie.ak.is.', filter='type:^A$ name:^apple')['dnsRecords']]) )
apple1 A 10.50.0.1
apple2 A 10.50.0.2
apple3 A 10.50.0.3 

In the example below we have created a Python script that logs on to a server, prints out all of its zones and then the first 10 records of a single zone. It assumes that we are running Men & Mice Web Service and Men & Mice Central on a server named 'central.demo'. It also assumes there is a zone named 'applepie.ak.is' on one of the servers:

import mmJSONClient
 
client = mmJSONClient.JSONClient()
client.Login(server='central.demo.', username='a_user', password='secret')
 
result= client.GetDNSZones()
print('\nTotal number of zones: ' + str(result['totalResults']))
for zone in result['dnsZones']:
    print(zone['name'])
 
myZone = myZone= 'applepie.ak.is.'
result = client.GetDNSRecords(dnsZoneRef= myZone, limit= 10)
print('\nRecords in ' + myZone)
for record in result['dnsRecords']:
    print(record['name'] + "\t" + record['ttl'] + "\t" + record['type'] + "\t" + record['data'])

Manipulating DNS Zones

Now let's do something a little bit more complicated. Let's create a zone and add some records to it. If we look at the list of SOAP API commands we can find AddDNSZone. AddDNSZone requires at least two arguments: a session argument we don't have to worry about since JSONClient will handle that for us, and a dnsZone argument which should be of type DNSZone. The DNSZone type requires at least two members (or arguments): name and type. The name argument is a string, and the type argument is an enumeration which can be Master, Slave, Hint, Stub or Forward. If the call is successful AddDNSZone will return a reference to the newly created zone.

We also have to specify where we want to create the zone. Notice that we don't have any possible DNS server reference when creating a zone. The reason is that each DNS server can have one or more views that in turn can contain a number of zones. In most cases DNS servers will only have one view, called the default view or the empty view (''). When referring to any object in the Men & Mice Suite, you can either use its globally unique identifier (GUID), that was returned from the Central Server or you can pass on a string that will uniquely identify the object, in this case using its name (for a more detailed description on how this works, see Referencing Objects in the SOAP API). To reference a default view on a server you can simply use its fully qualified name and add a double column at the end.

As an example, let's say we want to send a reference to the default view as an argument on the DNS server dnsserver1.demo. Here we can simply pass the string 'dnsserver1.demo.:' as a DNS view reference.

>>> zoneRef= client.AddDNSZone(dnsZone={'dnsViewRef':'dnsserver1.demo.:', 'name':'test1.com', 'type':'Master'} , saveComment='A zone created using Python')
>>> print(zoneRef)
{u'ref': u'{#4-#2054}'} 

Note that we also added a save comment when creating the zone as it is sensible to always add a comment when you are making changes to the system. Also notice the return value from AddDNSZone. As before it is a Python dictionary, this time with only the one member, 'ref', that contains a reference to the newly created zone. Now let's create few records:

>>> for i in range(100,105):
...     client.AddDNSRecord(dnsRecord={'dnsZoneRef':zoneRef['ref'], 'name':'test{0}'.format(i), 'ttl':'', 'type':'A', 'data':'10.50.0.{0}'.format(i), 'enabled':1})
... 

In this example, since I'm creating multiple records, I might instead use the SOAP command AddDNSRecords. AddDNSRecords will accept multiple records and update the zone just once instead of multiple times:

>>> client.AddDNSRecords(dnsRecords= [{'dnsZoneRef':zoneRef['ref'],  'name':'test{0}'.format(i), 'ttl':'', 'type':'A', 'data':'10.50.0.{0}'.format(i), 'enabled':1} for i in range(100,105)])
{'errors': [], 'objRefs': ['{#13-#9009}', '{#13-#9010}', '{#13-#9011}', '{#13-#9012}', '{#13-#9013}']}

As a final example, let's create a script that will check if all the PTR records for a DNS server are in order. The script will read through all the forward and reverse zones and report if there are any PTR records missing or if there are any orphaned PTR records:

import mmJSONClient
 
def qualifyName(recName, domainName):
    if len(recName) == 0:
        return domainName
    elif recName[len(recName) - 1] == '.':
        return recName + domainName
    return recName + '.' + domainName
 
mmServer    = 'central.demo.'
mmUser      = 'a_user'
mmPassword  = 'secret'
dnsServer   = 'b-centos6-32.demo.'
 
try:
    print('Checking reverse zone on server [{}] ...'.format(dnsServer))
    client = mmJSONClient.JSONClient()
    client.Login(server= mmServer, username= mmUser, password= mmPassword)
    viewRef = client.GetDNSView(dnsViewRef= dnsServer + ':')
    # Get all forward and reverse zones
    zoneList = client.GetDNSZones(filter= 'dnsViewRef:' + viewRef['dnsView']['ref'])['dnsZones']
    forwardZoneList = [z for z in zoneList if (not z['name'].endswith('in-addr.arpa.') and not z['name'].endswith('ip6.arpa.'))]
    revZoneList = [z for z in zoneList if z['name'].endswith('in-addr.arpa.')]
    print('Found {} forward-zone(s) and {} reverse-zone(s)'.format(len(forwardZoneList), len(revZoneList)))
  
    # Collect all PTR records as a map of ip -> [record, zone, data] 
    # and give the user a warning if there are duplicates
    ptrRecordsMap = {}
    for z in revZoneList:
        print('Scanning reverse-zone [{}]'.format(z['name']))
        try:
            records = client.GetDNSRecords(dnsZoneRef= z['ref'], filter= 'type:^PTR$')['dnsRecords']
        except Exception as e:
            print(e)
        for r in records:
            domain = qualifyName(r['name'], z['name'])
            label = domain.split('.', 4)
            ip = '{}.{}.{}.{}'.format(label[3], label[2], label[1], label[0])
            if ip in ptrRecordsMap:
                print(  ('Warning: Ignoring reverse record [{} in {} {}] for the IP address [{}]:\n'
                        + '\talready found [{} in {} {}]').format(
                              r['name'], z['name'], r['data'], ip
                            , ptrRecordsMap[ip][0], ptrRecordsMap[ip][1], ptrRecordsMap[ip][2]))
            else:
                ptrRecordsMap[ip] = [r['name'], z['name'], r['data']]
 
    # Collect all A records as a map of ip -> [record, zone] 
    # and give the user a warning if there are duplicates
    aRecordsMap = {}
    aMatchedRecordsMap = {} # contains all record that found a match
    for z in forwardZoneList:
        print('Scanning forward-zone [{}]'.format(z['name']))
        records= []
        try:
            records = client.GetDNSRecords(dnsZoneRef= z['ref'], filter= 'type:^A$')['dnsRecords']
        except Exception as e:
            print(e)
        for r in records:
            ip = r['data']
            if ip in aRecordsMap:
                print(  ('Warning: Ignoring the A record [{} in {} {}]:\n'
                        + '\talready found [{} in {} {}]').format(
                            r['name'], z['name'], r['data']
                            , aRecordsMap[ip][0], aRecordsMap[ip][1], ip))
            elif ip in ptrRecordsMap:
                # We already have this as a PTR record but do the data match?
                if qualifyName(r['name'], z['name']) != ptrRecordsMap[ip][2]:
                    print(  ('Warning: Record [{} in {} {}] has an incorrect reverse reference:\n'
                            + '\talready found a PTR record [{} in {} {}]').format(
                                r['name'], z['name'], r['data']
                                , ptrRecordsMap[ip][0], ptrRecordsMap[ip][1], ptrRecordsMap[ip][2]))
                aMatchedRecordsMap[ip] = [r['name'], z['name']]
                del ptrRecordsMap[ip]
            else:
                # the ip wasn't found in the reverse list but it might already been matched
                if ip in aMatchedRecordsMap:
                    print(  ('Warning: Ignoring the A record [{} in {} {}]:\n'
                            + '\talready found [{} in {} {}]').format(
                                r['name'], z['name'], r['data']
                                , aMatchedRecordsMap[ip][0], aMatchedRecordsMap[ip][1], ip))
                else:
                    aRecordsMap[ip] = [r['name'], z['name']]
 
    print ('\nFound {} missing PTR-Record(s)...'.format(len(aRecordsMap)))
    for ip in aRecordsMap:
        print ('{} in {} {}'.format(aRecordsMap[ip][0], aRecordsMap[ip][1], ip))
    print ('\nFound {} orphan PTR-Record(s)...'.format(len(ptrRecordsMap)))
    for ip in ptrRecordsMap:
        print ('{} in {} {}'.format(ptrRecordsMap[ip][0], ptrRecordsMap[ip][1], ptrRecordsMap[ip][2]))
except Exception as e:
    print(e)

Summary

The primary focus of the Men & Mice development team has always been to simplify the complex task of administering a DDI environment while still being flexible, as different environments can have very different needs. To achieve this flexibility the Men & Mice Suite provides rich set of commands accessible through a web service interface.

As of version 6.6 both SOAP/WSDL and JSON-RPC services are supported. Both of these services have their advantages but JSON-RPC is more light-weight than SOAP/WSDL and a better fit for many programming languages.

Topics: DNS, DHCP, Men & Mice Suite, DDI, DNS Zone, Web Services, JSON, SCRIPTING

An Introduction to the Men & Mice Web Service: SOAP API

Posted by Men & Mice on 8/29/14 9:51 AM

The Men & Mice Suite provides a large number of powerful features to manage DNS, DHCP and IP infrastructures of all sizes. All of these features are accessible through the Men & Mice GUI client or the web interface that come with the Suite.

An alternative to using the graphical tools, is to use the Men & Mice web service API to automate repetitive tasks. The web service has even been used to build new custom applications on top of the Men & Mice Suite. In this article, I will start by giving a brief overview of the API, followed by a short example to demonstrate how the service can be used.

Overview

The Men & Mice web service API first came out in 2008, and has been in constant development ever since. It uses the Simple Object Access protocol (SOAP) and has a Web Service Description Language (WSDL) definition that describes all operations that can be done. All new features in the Suite are implemented for the web service, so everything that can be done using the GUI tools can also be done through the API. The latest official API documentation can be found here. You can also see what operations are supported by your current release by connecting to the Men & Mice Central server that comes with the Suite (see the official API documentation for further details).

Examples of what users are doing with the Men & Mice web service include:
  • Search for DNS records in all zones that match a particular name.
  • Change the time to live (TTL) for all DNS records containing a certain pattern.
  • Construct a report for DHCP scope options and address pools, and e-mail to responsible personnel.
  • Automatically create DHCP reservations when deploying virtual machines.
  • Create DNS zones in bulk according to a template.
  • Manage a list of blacklisted DNS zones.
  • Run periodically a cleanup of IP addresses that haven't been seen on the network for some fixed time period.
  • Find the next free IP address, and create records and/or reservations through own web portals.

Numerous SOAP clients exist for different programming languages. For this article, we will be using the python programming language and the suds SOAP framework.

The first thing to do, when using the web service is to download the WSDL file from the Men & Mice Central server, that defines what SOAP commands are available for your release. The next step is to log into the system using the Login command to retrieve a session token that will be used to authenticate all subsequent calls to the service. Since the session token has to be used for all requests to the service, it becomes very repetitive to manually include it in all calls. We will, therefore, use a wrapper client, available from the Men & Mice website that will automatically append the token to all subsequent calls, once we have logged onto the system. Suds also requires a slight modification of the SOAP envelope in order to work with the Men & Mice web service, which will be handled by the client.

import suds
from soapCLI import mmSoap
 
try:
    cli = mmSoap(proxy="proxy.example.com",server="central.example.com",
             username="administrator", password="secret")
    cli.login()
except suds.WebFault as e:
    print "Error while logging into the Men & Mice suite: %s" % e.fault.faultstring

The same authentication model applies to users logged in through the web service, as to users using the GUI. The user needs to have permissions to work with a given resource (DNS records, zones, servers, users, etc). Special permissions are also needed to execute commands through the web server interface. The permissions can be granted by an administrator using a GUI, or the web service.

Error handling

The SOAP specification defines a format for messages containing error information and is used by the Men & Mice web service to notify a client when an operation fails. In some cases, we might also get partial failures. If we are for example deleting DNS records from a number of servers, then some of the DNS servers might be unreachable, while other servers successfully remove their corresponding records. In that case the operation will return a collection of error messages describing the errors.

Working with resources

All objects within the Men & Mice Suite, whether they are IP addresses, DNS zones, DNS records, DHCP scopes or any other type of resource, have a unique resource identifier that can be used to reference, retrieve and work with the resource. It is also possible to fetch a collection of items, for example DNS records and search the retrieved items for a specific pattern. Many of the SOAP commands also provide a filter input parameter, in order to filter what records to return from the service. Further details about references and filters can be found in the official Men & Mice API documentation .

Example usage

We will now demonstrate a simple, yet realistic example of how the API might be used.

The example demonstrates a scenario where we need to relocate a large number of websites hosted on different machines to a new web server. Instead of changing each DNS record manually by hand, we will automate the task by using a simple script.

We start by logging into the system using the before mentioned client available from the Men & Mice website. We then create an array of DNS records (arrayOfDNSRecordsToAdd) that will be used to store the new records that we want to add. We also create an array to store unique references to the records that we want to delete (dnsRecordsToRemove), after we have saved the newly created records.

try:
    cli = mmSoap(proxy=proxy,server=server,
             username=username, password=password)
    cli.login()
except suds.WebFault as e:
    print "Error while logging into the Men & Mice suite: %s" % e.fault.faultstring
    return False
    
arrayOfDNSRecordsToAdd = cli.create("ArrayOfDNSRecord")
dnsRecordsToRemove = cli.create("ArrayOfObjRef")

In order to get all DNS records, we need to fetch all DNS zones in the system using the GetDNSZones command. Once we have an object identifier for each zone (dnsZoneRef), we can use the GetDNSRecords SOAP command to retrieve all DNS records that are in the zone. Two filters are used. The first filter is used to make sure that only master zones are retrieved. The second filter is used to limit the results to records of type "A". If a record points to any of the servers we are migrating from (IP addresses 10.1.1.10, 10.1.1.11 or 1.1.1.1), then we add the record to the collection of records we want to remove and create a new DNS record pointing to the web server we are migrating to (IP address 10.0.0.1).

# Retrieve all master zones
(_, arrayOfDNSZone), (_, totalResults)  = cli.GetDNSZones(filter="type:Master")
if totalResults == 0:
    print "No DNS zones found."
    return True
for dnsZone in arrayOfDNSZone.dnsZone:
    # Fetch all A records from the zone 
    (_, arrayOfDNSRecord), (_, totalResults) = cli.GetDNSRecords(dnsZoneRef=dnsZone.ref, filter="type:^A$")
    if totalResults > 0:
        for dnsRecord in arrayOfDNSRecord.dnsRecord:
            if dnsRecord.data in ["10.0.0.10", "10.1.1.11", "1.1.1.1"]:
                # Modify records to point to new server
                dnsRecordsToRemove.ref.append(dnsRecord.ref)
                dnsRecord.ref = None
                dnsRecord.data = "10.0.0.1"
                arrayOfDNSRecordsToAdd.dnsRecord.append(dnsRecord)

Once we have created all the records, we add them to the system using the AddDNSRecords command. The command returns an array of record references for each new DNS record that has been added, along with an array of errors, if any. The references are used as a parameter to the GetDNSRecord command, that we use to get further information about the records.

The retrieved arrayOfDNSRecordRef will have the same record order as the arrayOfDNSRecordsToAdd that was passed to the SOAP command. If an error occurred when creating a particular record, then a "null" reference "{#0-#0}" will be returned for that record instead. The results could be used to make sure that any old records that correspond to the ones we were unable to add, will not be removed in the next step when we delete old records. We will, however, omit that step for this example and instead prompt the user to do a manual cleanup in case of any errors.

if len(arrayOfDNSRecordsToAdd.dnsRecord) > 0:
    # Create new records
    (_, arrayOfDNSRecordRef), (_, addRecordArrayOfErrors) = cli.AddDNSRecords(dnsRecords=arrayOfDNSRecordsToAdd, 
                                                                                saveComment='Modifying DNS records to point to host "10.0.0.1".')
    if len(arrayOfDNSRecordRef.ref) > 0:
        for recordRef in arrayOfDNSRecordRef.ref:
            if recordRef == "{#0-#0}":
                # Caused by a record that was not added due to some error
                continue
            try:
                print "Added record:", cli.GetDNSRecord(dnsRecordRef=recordRef)
            except suds.WebFault as e:
                print 'Unable to retrieve record with ref "%s" due to the following error: %s' % (recordRef, e.fault.faultstring)

If any errors came up while adding new records, then we instruct the user to do a manual clean up. Note that the error property from the AddDNSRecords response is "nillable". That means that it might not be present in the response. We therefore check if it has been set, before using it.

If no errors came up while adding new records, then we remove the old records using the RemoveObjects operation. The RemoveObjects command can be used to delete almost any object in the system, given that we have a resource reference to the object.

 

if hasattr(addRecordArrayOfErrors, "error") and len(addRecordArrayOfErrors.error) > 0:
    print """One or more errors occurred while adding records: %s.
 Old records will not be deleted. Please check manually and delete old records that were successfully migrated.""" % addRecordArrayOfErrors.error
    return False
else:       
    # No errors came up during migration, we delete the old records
    removeRecordErrors = cli.RemoveObjects(objRefs=dnsRecordsToRemove,
                                            saveComment="Removing records that now point to host %s." % addressTo)
    if len(removeRecordErrors) > 0:
       print "The following errors occurred while removing old records:", removeRecordErrors
       return False

The complete example can be downloaded here.

Summary

The Men & Mice web service API can be used to automate repetitive DNS, DHCP and IP address management tasks and can be used to write custom applications on top of the Men & Mice Suite. The service can be used by most common programming languages, and as demonstrated, is easy to use.

Topics: Men & Mice Suite, IPAM, DNS, DHCP, DNS Zone, Web Services, SCRIPTING

Take your DNSSEC with a grain of salt

Posted by Dora Vigfusdottir on 1/12/12 8:22 AM

Take your DNSSEC with a grain of salt

by Carsten Strotmann 
originally published at DNS Workshop.

DNSSEC has many useful properties. One is called 'Authenticated denial of existence'. This basically means that a DNSSEC validating DNS Server can prove that domain-names and resource records do not exist in the DNS.

But how does NSEC and NSEC3 work. And how to choose good values for NSEC3 salt and iterations?

Authenticated denial of existence

It requires some 'out of the box' thinking to understand how DNSSEC can prove things that do not exist. The solution is to list all the things (DNS Names and Resource Records) that do exist, and secure that list. If that list can be proven, then everything not listed must not exist. This property of DNSSEC prevents an attacker from sending fake negative DNS responses, like 'google.com -> NXDOMAIN', which could be used for denial of service attacks.

This list of things that do exist in a DNS zone is created by the NSEC (or NSEC3) records. These records build a linked-list from every domain name in a DNS zone, listing all resource record types for each name. Lets have a look at an example, the simple DNS zone 'myzone.example.com.'. Here is the plain DNS zone-file, without DNSSEC:

 

myzone.example.com.   3600 IN SOA  dns1.myinfrastructure.org.  hostmaster.myinfrastructure.org. (
              2011123001
              2h
              1h
              1000h
              1h )
myzone.example.com.     3600 IN NS dns1.myinfrastructure.org.
myzone.example.com.     3600 IN NS dns2.myinfrastructure.org.
myzone.example.com.     3600 IN MX 10 mail.myinfrastructure.org.
www.myzone.example.com. 3600 IN A 192.0.2.100
www.myzone.example.com. 3600 IN AAAA 2001:db6:100::80

The zone including the NSEC records will look like this (I'm omitting all RRSIG signature-records and DNSSEC key records here, because the focus is on the NSEC records):

 

myzone.example.com.     3600    IN SOA  dns1.myinfrastructure.org. hostmaster.myinfrastructure.org. (
                                        2011123001 ; serial
                                        7200       ; refresh (2 hours)
                                        3600       ; retry (1 hour)
                                        3600000    ; expire (5 weeks 6 days 16 hours)
                                        3600       ; minimum (1 hour)
                                        )

                        3600    NS      dns1.myinfrastructure.org.
                        3600    NS      dns2.myinfrastructure.org.
                        3600    MX      10 mail.myinfrastructure.org.
                        3600    NSEC    www.myzone.example.com. NS SOA MX NSEC
www.myzone.example.com. 3600    IN A    192.0.2.100
                        3600    AAAA    2001:db6:100::80
                        3600    NSEC    myzone.example.com. A AAAA NSEC

In this simple zone, we have two domain names (myzone.example.com. and www.myzone.example.com.), and each domain name owns one NSEC record, listing all the resource record types that exist for that name, and pointing to the next domain name in the zone (in lexicographical order). The last NSEC record wraps around and points to the very first domain name in the zone.

 

Walking a DNS zone

Using NSEC is relatively simple, but it has a nasty side-effect: it allows anyone to list the zone content by following the linked list of NSEC records. This is called 'zone walking'. The 'ldns' library contains an tool called 'ldns-walk' that can be used to list all records inside a DNSSEC signed zone that uses NSEC:

 

$ ldns-walk paypal.com
paypal.com.     paypal.com. A NS SOA MX TXT RRSIG NSEC DNSKEY TYPE65534 
3pimages.paypal.com. A RRSIG NSEC 
_dmarc.paypal.com. TXT RRSIG NSEC 
ym2._domainkey.paypal.com. TXT RRSIG NSEC 
3rdparty._spf.paypal.com. TXT RRSIG NSEC 
3rdparty1._spf.paypal.com. TXT RRSIG NSEC 
3rdparty2._spf.paypal.com. TXT RRSIG NSEC 
pp._spf.paypal.com. TXT RRSIG NSEC 
_autodiscover._tcp.paypal.com. SRV RRSIG NSEC 
accounts.paypal.com. CNAME RRSIG NSEC 
active-history.paypal.com. A RRSIG NSEC 
active-www.paypal.com. A RRSIG NSEC 
adnormserv.paypal.com. A RRSIG NSEC 
adnormserv-dft.paypal.com. A RRSIG NSEC 
adnormserv-phx.paypal.com. A RRSIG NSEC 
adnormserv-slc-a.paypal.com. A RRSIG NSEC 
adnormserv-slc-b.paypal.com. A RRSIG NSEC 
....

For some DNS zones, this is an issue. The NSEC3 record option in DNSSEC solves this by creating the linked list using hashed domain-names, instead of clear-text domain names.

It is important to know that NSEC3 also enables a new function in DNSSEC, called 'opt-out'. With 'opt-out', it is possible to 'jump-over' (delegation) domain names in a zone where the child zone itself is not DNSSEC signed (it does not increase the security to provide a secure delegation in case the child itself is insecure). Zone administrators can decide to deploy NSEC3 to prevent zone walking, or to use 'opt-out', or both. Top-Level zone administrators choose NSEC3 because of this 'opt-out' function. This needs to be taken into account when looking at the NSEC3 parameters of DNSSEC signed zones.

In this blog post I will look into the NSEC3 parameters that prevent 'zone-walking', and I will not discuss 'opt-out'.

NSEC3 cannot prevent 'zone-walking' entirely, but it can make 'zone-walking' more expensive for the attacker. There are two parameters in NSEC3 that can be set by the administrator of a zone to fine tune the amount of work required on the attacker side to 'walk' a DNSSEC signed zone with NSEC3 records: the salt and the number of iterations.

 

The salt

NSEC3 is using a hashing algorithm to disguise the real DNS domain names used. It is impossible to recreate the original domain names from the hash name. Here is our zone, but now using NSEC3 (again, RRSIG and DNSKEY records have been removed):

 

myzone.example.com.     3600    IN SOA  dns1.myinfrastructure.org. hostmaster.myinfrastructure.org. (
                                        2011123001 ; serial
                                        7200       ; refresh (2 hours)
                                        3600       ; retry (1 hour)
                                        3600000    ; expire (5 weeks 6 days 16 hours)
                                        3600       ; minimum (1 hour)
                                        )
                        3600    NS      dns1.myinfrastructure.org.
                        3600    NS      dns2.myinfrastructure.org.
                        3600    MX      10 mail.myinfrastructure.org.
                        0       NSEC3PARAM 1 0 10 1A2B3C4D5E6F
6HAUGENGIFFS98J25KBEASD720PGAN36.myzone.example.com. 3600 IN NSEC3 1 0 10 1A2B3C4D5E6F (
                          BE61EEUDCS4VQO71L3LFJMRKEL16S534 A AAAA )
www.myzone.example.com. 3600    IN A    192.0.2.100
                        3600    AAAA    2001:db6:100::80
BE61EEUDCS4VQO71L3LFJMRKEL16S534.myzone.example.com. 3600 IN NSEC3 1 0 10 1A2B3C4D5E6F (
                          6HAUGENGIFFS98J25KBEASD720PGAN36 NS SOA MX NSEC3PARAM )

The first NSEC3 record is for the name "www.myzone.example.com.", while the last one is for the zones apex ("myzone.example.com."). The NSEC3 records in a DNSSEC signed zone are not in the proximity of the domain names they are securing (but the list of record types can give a hint).

An attacker needs to create a rainbow table to be able to 'walk' a NSEC3 zone. A rainbow table lists precomputed hashes for common domain names in that zone. The attacker will take the hashed domain names returned in an NSEC3 resource record and will compare the hash with the values in the rainbow table. If the hash matches one entry in the table, the clear text of the domain name is found. The alternative approach would be to brute force try all possible domain names and send queries to the zone. However this approach might be detected if the DNS queries towards the authoritative DNS servers are monitored (and you monitor the queries towards your DNS, do you?).

Using a rainbow table the attacker can first collect all hashed domain names by following the NSEC3 linked-list, and then try to reverse the hashes using the rainbow table in the privacy of his own environment.

The input to the hash is always the full qualified domain name (like 'www.myzone.example.com.'), so the address records pointing to a web-service "www" in two different zones will hash into different hash values. An attacker will need to create a dedicated rainbow table for each DNS zone. But once the table is calculated and working, the attacker can re-use the table for every subsequent scan.

To prevent the re-use (or even the first use of a rainbow table), there is the salt. The salt is a random, hexadecimal string that is appended to the domain name before applying the hash function to the name. The salt in the example above is '1A2B3C4D5E6F' (the last parameter in the NSEC3PARAM record, and the 4th parameter of every NSEC3 record). This salt is appended to the domain name 'www.myzone.example.com.1A2B3C4D5E6F' (but the domain name would be in wire format, and the hash in binary, not hexadecimal!) and then the hash function (today SHA1) is applied. The tool 'ldns-nsec3-hash' can be used to hash a domain name on the command line:

$ ldns-nsec3-hash -t 10 -s 1A2B3C4D5E6F  www.myzone.example.com 
6haugengiffs98j25kbeasd720pgan36.

The salt value is public (it is in the NSEC3PARAM record in the zone, and in every NSEC3 record). It must be public, because a validating DNS server must know the salt value to be able to validate the NSEC3 records coming in a negative DNS answer. Because it is public, attackers can fetch the value from the zone to generate a rainbow-table. Therefore, the salt needs to be changed from time to time.

Whenever the salt is changed in a zone, the attacker needs to throw away any pre-computed rainbow-table for the zone and start re-creating the table from scratch. If the salt is changed in intervals shorter than the time it takes to compute a reasonable large rainbow-table, it makes 'zone walking' impossible.

RFC 5155 that defines the NSEC3 record recommends to change the salt whenever a zone is signed. Because a new salt will change all NSEC3 records, and all RRSIG records for the NSEC3 records, this is quite some overhead for a large zone. The RFC draft 'draft-ietf-dnsop-rfc4641bis-08' recommends to change the salt whenever the zone signing key is rolled (because that triggers re-generation of all signatures anyway).

The salt can be up to 510 hexadecimal characters, but in practice, using 8-16 hexadecimal characters is a good value (32-64 bit). It is recommended to automate the salt generation: whenever a new zone signing key (ZSK) is generated, there should be a new salt generated.

One possible way to generate the salt is to use 16 characters out of the SHA1 hash of the current date and time:

$ date | sha1sum | cut -b 1-16
784d739287d3336

If the salt is generated from a cron script, using the date as the input to the hashing function might be predictable by an attacker. In that case, using 512 byte of randomness can be used as an alternative:

 

$ head -c 512 /dev/random | sha1sum | cut -b 1-16
7ac59a6dd87fa477

It is possible to use NSEC3 without a salt. In the signing command, and in the NSEC3PARAM resource record, an empty salt value is written with a single dash '-' (NSEC3PARAM 1 0 0 -). Not using a salt still protects again basic zone walking, but it allows an attacker to create and re-use a rainbow table. The 'com' gTLD is using an empty salt:

 

$ dig com nsec3param
; <<>> DiG 9.7.3 <<>> com nsec3param
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1835
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 0

;; QUESTION SECTION:
;com.                           IN      NSEC3PARAM

;; ANSWER SECTION:
com.                    86400   IN      NSEC3PARAM 1 0 0 -

;; Query time: 154 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Dec 30 14:03:48 2011
;; MSG SIZE  rcvd: 262

 

The iterations

The second parameter important for the NSEC3 hash function is the number of iterations (3rd field in the NSEC3 and NSEC3PARAM records). As computer become more powerful over time, it might be possible to calculate rainbow tables in a short amount of time. The number of iterations in NSEC3 controls how often the result of the first hash operation is hashed again (so even with iterations=0, the domain names are hashed once). By increasing the number if iterations, calculating a rainbow table can be made more expensive. But it also makes sending negative answers more expensive for authoritative DNS servers hosting the zone, as well as validating these answers on the receiver side. The number of iterations must be carfully balanced, a too high number can invite denial-of-service attacks agains the zones authoritative servers.

The current default value for NSEC3 iterations in BIND 9.8 is 10 iterations. the number of iterations is always a compromise between security and CPU usage. The default in the BIND signing tool is a good value for most zones, but every administrator should evaluate if the default is also good for her/his zones.

The number of iterations have a dependency on the size of the smallest zone-signing key in the zone (see RFC 5155, Section 10.3). These are the upper limit values (do not use higher values for the iteration field!):

 smallest ZSK Key Size in zone  max possible NSEC3 iterations  recommended by RFC 4641bis 
1024 150 100
2048 500 330
4096 2500 1600

NSEC3 records that use too high iteration values will be seen as 'insecure' by a validating DNSSEC resolver! (see RFC 5155, 12.1.4). RFC 4641bis recommends 2/3 of the maximum value. For a 1024 bit ZSK this would be 100 iterations.

 

Conclusion

 

  • If you wish to prevent 'zone walking', then sign your zone with NSEC3. If you don't mind (DNS data is public data), use plain NSEC.
  • if your domain is a 'important enough target' (whatever that might be), and there is risk that someone creates a rainbow-table for your zone, schedule a change of the NSEC3 salt parameter every time you roll your zone signing key.
  • from time to time re-evaluate the number of hash iterations used in your NSEC3 signed zone, based on the advances in computing power that might be available to a potential attacker.

 

References:

 

Thanks to Peter Koch, Alan Clegg and Miek Gieben for valuable input and answering my questions.

Topics: DNSSEC, DNS, Men & Mice, DNS Zone

Why follow Men & Mice?

The Men & Mice blog publishes educational, informational, as well as product-related material for everyone and anyone interested in IP Address Management, DNS, DHCP, IPv6, DNSSEC and more.

Subscribe to Email Updates

Recent Posts