The Men & Mice Blog

Supervising BIND 9

Posted by Men & Mice on 1/21/15 2:59 PM

BIND 9 is a mature piece of software, and compared with its predecessors BIND 4 and BIND 8 it is noticeably more stable and secure. One reason for this is the "Design by contract" programming style used by the BIND 9 team; as a result, BIND 9 is very particular about the data it consumes, and about its own internal data structures. Once BIND 9 encounters an unexpected state in its data structures, it terminates the DNS server process rather than continue running with bad data (and thus potentially compromise security).

While this behavior has clear advantages in terms of security, it can adversely affect service uptime - BIND 9 had several incidents in the past years where BIND 9 terminated because of issues inside the code or data structures, such as "BIND 9 Resolver crashes after logging an error in query.c". And for all its security benefits, an end user unable to reach Facebook may not be terribly understanding in the event of an outage.

The real issue, however, is not that BIND terminates when it comes across bad data, but rather that the process cannot automatically restart after the fact; there is no "supervisor" process in BIND 9.

Some operating systems have a built-in solution: MacOS X has launchd, and the BIND 9 version Apple delivers with the OS is automatically restarted should it terminate unexpectedly. Solaris has SMF (Service Management Facility), and BIND 9 can be integrated into SMF. Recent versions of Ubuntu, RedHat Enterprise, SuSe Enterprise, and Fedora now all use systemd, which can also monitor processes and restart them if needed.

But for Unix and Linux operating systems that do not ship with a process supervisor solution, supervisord is a strong alternative, with the added benefit of being relatively easy to install and configure. Supervisord comes as a package with many Linux distributions, and also works on BSD distributions.

The configuration below is intended for RedHat 6, but should require only minor tweaks to run on other Unix systems as well.

Installation

Supervisord is written in Python (2.4 - 2.7) and can be installed from source (where we have to download and install all dependencies) or with the help of setuptools, which takes care of downloading and installing dependencies (Meld3 and ElementTree).

Full Installation instructions can be found at [http://supervisord.org/installing.html]

Automatic Installation

‣ download "setuptools" from [https://pypi.python.org/packages/source/s/setuptools/setuptools-9.1.tar.gz]

 shell> tar xfz setuptools-9.1.tar.gz 
shell> cd setuptools-9.1 
root-shell> python setup.py install

Once setuptools have been installed, run the following command to install Supervisor and all required dependencies:

 root-shell> easy_install supervisor 

Manual Installation

Supervisor and its dependencies can also be installed manually.

‣ download "setuptools" from [https://pypi.python.org/packages/source/s/setuptools/setuptools-9.1.tar.gz]

 shell> tar xfz setuptools-9.1.tar.gz 
shell> cd setuptools-9.1 
root-shell> python setup.py install

‣ download "Meld3" from [http://www.plope.com/software/meld3/meld3-0.6.5.tar.gz]

 shell> tar xfz meld3-0.6.5.tar.gz 
shell> cd meld3-0.6.5 
root-shell> python setup.py install

‣ download "ElementTree" from [http://effbot.org/media/downloads/elementtree-1.2.6-20050316.tar.gz]

 shell> tar xfz elementtree-1.2.6-20050316.tar.gz 
shell> cd cd elementtree-1.2.6-20050316 
root-shell> python setup.py install

‣ download "Supervisor" from [https://pypi.python.org/packages/source/s/supervisor/supervisor-3.1.3.tar.gz]

 shell> tar xfz supervisor-3.1.3.tar.gz 
shell> cd supervisor-3.1.3 
root-shell> python setup.py install

Installing startscript and sysconfig

‣ download the startscript from [https://raw.githubusercontent.com/Supervisor/initscripts/master/redhat-init-jkoppe] and place it in /etc/init.d/supervisord

 root-shell> cp redhat-init-jkoppe /etc/init.d/supervisord 
root-shell> chmod +x /etc/init.d/supervisord

‣ download the 'sysconfig' file from [https://raw.githubusercontent.com/Supervisor/initscripts/master/redhat-sysconfig-jkoppe] and place it in /etc/sysconfig/supervisord

 root-shell> cp redhat-sysconfig-jkoppe /etc/sysconfig/supervisord 

Installing Bind 9 from Men & Mice repositories

‣ download the BIND 9 RPM from [http://support.menandmice.com/download/bind/linux/redhat/6.x/]<arch>/<version>/

 root-shell> yum install ISCBIND-<version>-<flavor>RHL<arch>.rpm 
root-shell> mkdir /var/named 
root-shell> useradd -d /var/named -r named 
root-shell> chown -R named: /var/named

‣ create a BIND 9 configuration file '/etc/named.conf'

options { directory "/var/named"; dnssec-validation auto; }; 

‣ create an 'rndc' configuration

root-shell> rndc-confgen -a 

‣ verify the configuration

root-shell> named-checkconf -z 

A basic configuration file for BIND 9 "named"

Below is my basic /etc/supervisord.conf configuration file for one service, the BIND 9 DNS Server:

 [unix_http_server] 
file = /tmp/supervisor.sock 
chmod = 0777 
chown= nobody:nobody 

[rpcinterface:supervisor] 
supervisor.rpcinterface_factory = supervisor.
rpcinterface:make_main_rpcinterface 

[supervisorctl] 
serverurl=unix:///tmp/supervisor.sock 
[supervisord] 
logfile = /var/log/supervisord.log 
logfile_maxbytes = 10MB 
logfile_backups=10 
loglevel = info 
pidfile = /var/run/supervisord.pid 
identifier = supervisor 
directory = /tmp 

[program:named] 
command=/usr/sbin/named -u named -f 
process_name=%(program_name)s 
numprocs=1 
directory=/var/named 
priority=100 
autostart=true 
autorestart=unexpected 
startsecs=5 
startretries=3 
exitcodes=0,2 
stopsignal=TERM 
stopwaitsecs=10 
redirect_stderr=false 
stdout_logfile=/var/log/named_supervisord.log 
stdout_logfile_maxbytes=1MB 
stdout_logfile_backups=10 
stdout_capture_maxbytes=1MB

Starting supervisord

With the configuration file in place, we can start supervisord. Make sure that BIND 9 is not started or you will end up with two instances of the BIND 9 server running, which isn't recommended. Also make sure that supervisord will be started on reboot of the server, either through a startscript or other means. Note that the supervisord packages bundled with Linux distributions install a startscript.

 root-shell> /etc/init.d/supervisord start 
root-shell> rndc status 
version: 9.9.6-P1 <id:3612d8fb> 
number of zones: 98 
debug level: 0 
xfers running: 0 
xfers deferred: 0 
soa queries in progress: 0 
query logging is OFF 
recursive clients: 0/0/1000 
tcp clients: 0/100 

server is up and running 
root-shell> ps -ef 
[...] 
root 10906 0.0 2.5 209096 12988 ? Ss 19:55 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf 
named 10908 0.7 1.6 44292 8112 ? S 19:55 0:00 /usr/sbin/named -u named -f 
root 10910 0.0 0.2 110228 1156 pts/0 R+ 19:55 0:00 ps aux 

root-shell> supervisorctl 
status 
named RUNNING pid 10908, uptime 0:03:19 
root-shell> chkconfig --add supervisord 
root-shell> chkconfig supervisord on

Great, supervisord has started, and it also started the BIND 9 process "named". DNS is working now.

Simulating a BIND 9 crash

To simulate a BIND 9 crash, we "kill" the BIND 9 named process:

root-shell> killall -9 named 

Supervisord should detect that the running BIND 9 process has terminated, and start a new one. DNS is still up and running.

Controlling supervisord

Supervisord can be controlled from the command line using the supervisorctl command. A list of all a control commands can be found with "help", and a description of each command with "help command":

 shell> supervisorctl help 
default commands (type help ): 
===================================== 
add clear fg open quit remove restart start stop update 
avail exit maintail pid reload reread shutdown status tail version 

shell> supervisorctl help status 
status Get all process status info. 
status Get status on a single process by name. 
status Get status on multiple named processes. 

shell> supervisorctl status named 
RUNNING pid 25770, uptime 0:00:12 

shell> supervisorctl stop named  named: stopped 

shell> supervisorctl start named 
named: started

Now, whenever there is a triggered assertion error in the code BIND 9 will terminate, but supervisord will bring it back from the dead. Your DNS service stays up, and your users and customers stay happy.

Read the supervisord documentation on how to setup event notifications, so that you get an e-mail notification should BIND 9 restart (should the outage be caused by a security vulnerability you might want to report it to bind9-bugs@isc.org as well).

Of course supervisord can be used to restart other processes as well, including other types of DNS Servers (NSD, Unbound, dnsmasq ...).

Topics: DNS, Linux, BIND, BIND 9, Supervisord, Red Hat

DNSSEC & DANE – E-Mail security reloaded

Posted by Men & Mice on 9/9/14 8:06 AM

Conventional TLS/SSL for SMTP is flawed and the x509 certificate business is a pain

Get a 35 minute crash course in "DANE" style securing x509 certificates using DNSSEC secured DNS using BIND 9 and the Postfix mail server from Mr. Carsten Strotmann from the Men & Mice Services team.

Carsten also answers the following questions:

  • Does the Men & Mice Suite support DANE & TLSA?
  • For DANE I need a DNS hosting provider that supports DNSSEC, where can I find one?
  • What impact will DANE have on the current certificate business?

Take a look!


 

Topics: DNSSEC, DNS, DDI, BIND 9, DANE

Delve deep into DNSSEC

Posted by Men & Mice on 3/28/14 8:49 AM

By Mr. Carsten Strotmann, one of Men & Mice experts.

BIND 9.10 is the new version of the BIND 9 DNS server from ISC (not to be confused with BIND 10, which is a different DNS server product). We will report in a series of articles about the new features in BIND 9.10. The first beta version of BIND 9.10 was released this week and can be found at ftp://ftp.isc.org/isc/bind9/9.10.0b1/.

BIND 9.10 contains a new command-line tool to test DNSSEC installations. The tool is called delve and it works very much like the well-known dig, but with special DNSSEC validation powers.

delve checks the DNSSEC validation chain using the same code that is used by the BIND 9 DNS server itself. Compared with the DNSSEC testing function in dig +sigchase, delve is much closer to what really happens inside a DNS server.

1.1 A simple lookup

Without extra arguments, delve will query the local DNS server (taken from /etc/resolv.conf) for an IPv4-Address record at the given domain name. It tries to validate the answer received, prints the result of the validation, the requested data and the RRSIG Record (DNSSEC signature) used to verify the data.

1.2 pretty-printing

As with dig, resource record types and network classes can be given in almost any order on the commandline. The switch +multi (for multiline) enables pretty printing; human readable output that is neatly formatted for a 78 column screen.

and IPv6

1.3 tracing DNSSEC validation

delve comes with a set of trace switches that can help troubleshoot DNSSEC validation issues. The first switch, +rtrace, prints the extra DNS lookups delve performs to validate the answer:

In this example, in addition to the MX-Record (Mail-Exchanger) Record, the DNSKEY record (DNSSEC public key) and the DS record (Delegation signer) for dnsworkshop.org, as well as the DNSKEY and DS records for ORG and the DNSKEY for the root-zone "." have been requested. The trust-anchor for the Internet Root-Zone is compiled into delve and acts as the starting trust anchor for the validation.

The switch +mtrace prints the content of any additional DNS records that have been fetched for validation.

+vtrace prints out the DNSSEC chain of validation:

delve is a very useful tool, not only for BIND 9 admins, but for everyone who needs to troubleshoot and fix DNS- and DNSSEC related issues.

Topics: DNSSEC, IPv6, BIND 9

BIND 9 Code Quality

Posted by Men & Mice on 1/23/14 5:35 AM

By Mr. Carsten Strotmann, one of Men & Mice experts.

BIND 9 and how a security issue demonstrates quality

Recently ISC issued a security warning (CVE-2014-0591) for several BIND versions.

The issue was that BIND 9 detects wrong data while working on NSEC3 records, and because the data is wrong, it opts to terminate itself instead of working with the wrong data (which could expose more serious security issues, esp. when handling DNSSEC data).

Shane Kerr of ISC described this behavior of BIND in the blog post "BIND 9′s Security Record": "The manner in which BIND 9 reacts to software bugs is to terminate. While unpleasant for administrators, the idea is to avoid the system running in an invalid state and causing more damage."

ISC's Michael McNally gave some background information on the security issue on the BIND users mailing list. The security issue has been caused by a change in the fundamental operating system library, the "libc". The implementation of the memcpy function has been changed in a recent update of the glibc library used on Linux systems. This change of implementation has triggered the bug to become visible. So far, the same bug has not been seen on other operating systems, or with other libc implementations. However, that does not mean that these systems are safe, just that the security issue does not show (but might still be there).

I'm happy about how BIND 9 handles this issue (terminating instead of ignoring the issue). This way the administrator notices (one hopes) and updates to a fixed version of BIND 9  and as binary installer packages for RedHat, Debian and Solaris from Men & Mice.

What scares me is all the other software out there (open source or commercial) that might be affected by this bug, but does not have the security net that BIND 9 has.

There could be similar security issues lurking in other software products. Stay vigilant! Monitor your servers.

As developers, we should scan our code for this error pattern (memcpy vs. memmove).

Topics: BIND 9, Security

Why follow Men & Mice?

The Men & Mice blog publishes educational, informational, as well as product-related material for everyone and anyone interested in IP Address Management, DNS, DHCP, IPv6, DNSSEC and more.

Subscribe to Email Updates

Recent Posts