DNS

DNS Management and Operations

Dns requests make up about 1/3 - 1/2 of all support requests. On the one hand, they are very easy requests to fulfill - but on the other hand, it is important to understand exactly how dns works at johncompanies, and how to repair or rebuild it. Dns is one of two things (the firewall being the other) that could destroy the business forever if it fails and is not solved very quickly.

JohnCompanies runs three dns servers:

ns1c.johncompanies.com  69.55.225.225
ns2c.johncompanies.com  69.55.230.3
ns3c.johncompanies.com  69.55.229.3

All of these are jailed FreeBSD systems - just like any other FreeBSD server instance we would sell to a customer. This means neither no server is its own purpose-built dedicated physical machine.

ns1c is a FreeBSD jail that lives on mail.johncompanies.com

ns2c is a FreeBSD jail that lives on jail18

ns3c is a FreeBSD jail that lives on nat2 at i2b

All nameservers were built exactly according to the sections below. A notable reference:

http://www.kozubik.com/docs/bind9_freebsd4.txt

A simple FreeBSD installation was used, then named.conf was dropped into /etc/namedb, and all of the zone files were dumped into /etc/namedb/s

This means that reconstruction due to a failure should be very easy. Since ns1c and ns2c are both backed up nightly, all that needs to be done is a new jailed system created and assigned the proper nameserver IP, bind loaded according to that document, and then the named.conf and the zone files taken from the most recent backup.

It should be noted that ns2c & ns3c have no primary records on it - all they do is slave zones from ns1c so in the case of a ns2c/ns3c failure, the only file you need is named.conf from ns1c. Ns2c & ns3c are automatically updated with zones from ns1c 2x a day on the 12’s (cron runs usr/local/sbin/sendzones.pl on ns1c and a few minutes later /etc/namedb/createzones.pl runs on ns2c & ns3c) so you should never have to touch it. If the replication process fails, an email will be sent to support- the problem is almost always on ns1c.

We interface with these three nameservers as if they were totally seperate, standalone servers. That is to say, when using and administering them, no concession at all is given to the fact that they are actually jailed systems running on some other underlying machine. You ssh to them in the screen window mentioned above just like any other system.

The configuration is very, very simple. Nothing is running on these systems except for bind - nothing else at all. The configuration file is /etc/namedb/named.conf, and consists simply of zone records that look like:

// col00005
zone "burnscustom.com" {
        type master;
        file "/etc/namedb/s/burnscustom.com";
        };

in the case of a master entry, or:

// col00023
zone "spews.net" {
        type slave; masters { 198.78.66.18; };
        file "/etc/namedb/s/spews.net";
        };

in the case of a slave entry. Very very simple. Note that both entries are exactly 6 lines of data with one blank line after/before it before the next zone entry- there shouldn’t ever be any more than 6 lines per definition.

The comment field at the beginning of every zone record in /etc/namedb/named.conf is the customer number. Additional/new zones should be added to the end of a customer’s current zone list.

In the case of slave zones, no other action needs to be taken other than entering the zone into named.conf, and HUPping the nameserver - the zone file will automatically be created in the /etc/namedb/s directory.

In the case of a master record, a new file needs to be created to match the filename in the record. By default, all new records should look like this:

$TTL    43200
@       IN      SOA     ns1c.johncompanies.com. dns.johncompanies.com.  (
                        2005062302
                        3600
                        900
                        3600000
                        3600 )
        NS      ns1c.johncompanies.com.
        NS      ns2c.johncompanies.com.
        MX      10      mail.echosoft.com.

echosoft.com.   A       69.55.226.59
*.echosoft.com. A       69.55.226.59

Notice how, by default, there is simply a wildcard entry for the domain ? Almost all of our domains are like that, and we do that by default. Another thing we do by default is make the primary MX record be mail.thatdomain.com.

This requires more explaining. If a new customer comes to us with a domain to put into the dns server, the zone needs to be entered into named.conf as shown above. Then, a new file like the one shown above needs to be created, and their domain name put into it - one line for the domain, and a second line for the wildcard entry. Then, we set the primary MX record to mail.thatdomain.com.

Obviously all of this only applies to master domains, where we do the primary dns for the customer - if we slave the domain from the customer, we have no input in this, and we simply suck over whatever they give to us.

Tools we use that make DNS management easier:

When adding any number of new primary domains, there is an interactive script to make this process easier:

createmaster [customerID] [ip]

Optional arguments to it are the customerID and the IP the domains will resolve to. If not given, the script will ask for them. The script will ask for each domain to be added, one domain per line. To finish, just enter a blank line. It will ask for confirmation that the domains read in are all correct and if confirmed, it will create zone files (in the format listed above, Note: if a zone file already exists, it will notify you and give you the option to overwrite or abort writing that zone file.) and echo to the screen the zone definitions to be added to named.conf. Those definitions should be copied and pasted into the conf file (if the customer has existing zones, the new ones should be added at the end of the customer’s existing zones). Once the new zones are in the conf, issue

rr 

which will tell bind to reload zones (rr is a wrapper for rndc reload). You should test to see if the new zones are working properly:

ns1c# host zigdon.org 69.55.225.225 

If the customer has asked for special entries (CNAME, MX, etc) you’ll have to manually edit the zone files. By default, they will look like above with MX set to mail.domain.tld

When adding new slave domains, there is also an script to make this process easier:

createslave [customerID] [ip]

Optional arguments to it are the customerID and the IP the domains will slave from. If not given, the script will ask for them. The script will ask for each domain to be added, one domain per line. To finish, just enter a blank line. It will ask for confirmation that the domains read in are all correct and if confirmed, it will (in the format listed above) and echo to the screen the zone definitions to be added to named.conf. Those definitions should be copied and pasted into the conf file (if the customer has existing zones, the new ones should be added at the end of the customer’s existing zones). Once the new zones are in the conf, issue rr which will tell bind to reload zones and ask their dns server for the zone info. You should test to see if the new zones info was received by looking at the contents of the new zone file:

ns1c# cat zigdon.org

If the the file is not found, tell the customer to HUP their dns server.

If a customer requests a change to an existing zone, make the changes to the zone file, and increment the serial by 1 digit. In the example above, the serial is 2001121102. After you’ve saved the file, issue

rr <domain>

That will reload the zone with the new info AND send an update to ns2c. If you forget to increment the serial, the update will not make it to ns2c and that will cause mixed info to be fed when zone lookups for that domain are performed. In that case just increment and rr the domain.

listdomains <customerID>

will list domains belonging to a customerID (per what’s contained in the conf)

1. If it is not already clear, the entries in named.conf are in number order of their customer number. So when adding a new entry, search for the customer number first - if there are already entries for that customer, then put it at the end of those entries. If there are not, make sure it gets placed in named.conf in order, according to the customer number.

2. when you run rr – which tail’s /var/log/messages as a sanity check to make sure nothing untoward got loaded into the config, and that it really did get reloaded. Seeing a lot of failed slavings is normal. Seeing an error related to the zone you just entered is what you’re looking for.

3. If bind ever dies - just `ps auxw` and make sure no named process is running, and then just run:

/usr/sbin/named

Since ns1c/ns2c are monitored by castle access, you will be notified immediately if this happens, and if it happens at non-peak hours, the two minutes total that it should be down will barely be noticed.

4. We must do RDNS for our customers. We set it up for them, when they ask - we do not delegate our ips

IMPORTANT: Be very careful with ns1c - it is the lifeblood of the entire business, and a loss of service on it is one of the few things that could permanently destroy the business. Think twice before running any script that could possibly overwrite any files in /etc/namedb/s, and double check all command lines that you run on this server.

If you’re ever lost, go to isc.org and subscribe to bind9-workers

To update RDNS, edit the appropriate file:

ns1c# vi 69.55.234.rev

Increment the serial by 1, then reload as follows:

ns1c# rr 234.55.69.in-addr.arpa

There are a couple other very helpful tools:

vr [domain]

will open the zone file for editing in vi and after exiting reload the zone. Equivalent to: vi [domain]; rr [domain]

listmasters

this will list all the zones for which we’re acting as a master

ttlswap <cid> <ttl>

this will change the ttl’s for all zones for a given cid. It will prompt before changing each zone.

ptr <ip>

this will open up the appropriate zonefile for editing an IP’s PTR record. After saving the file, it will reload the zone.

rmdomains <cid>

this will remove the zonefiles for any domain found for the given cid inside named.conf. This only removes the zonefiles, it does nothing to named.conf. So you still have to edit named.conf to remove the zone definitions manually, then reload bind (rr).

troubleshooting slaving issues

If there's an issue where we're not resolving a domain which we are slaving, the likely cause is the domain expired and we never got updated zone info from the master. To confirm this, make sure we're slaving from the right server, then rm the zone file and rr the domain and see if we get a fresh zonefile.

If occasionally there is an issue where, for instance we're not getting a new zone file and the customer's setup is good, we've seen occasions where no amount of rr [ZONEFILE] fixes the problem or rr, and you just have to:

rndc stop
ps auxw|grep named (make sure its down)
named

to restart the nameserver, this will clear up that issue.

Restore deleted domain

In the event a domain (zone file or done definition inside named.conf) is deleted, restoring is easy. We make backups of the mail server daily, included in that are bind config files and domains. These backups may be found on backup1:

/data/www/daily

and are organized by date. So diggind in, we will find all the files we're looking for here (for example):

ls /data/www/daily/12-12-28/mnt/data1/ns1c-dir/etc/namedb/
bind.keys                                                 named.orig
copys.sh                                                  named.prev
createzones.pl                                            named.root
k                                                         named.root.orig
make-localhost                                            named.stats
named.conf                                                named.tar
named.conf.back-200908140748                              ns2c.tar
named.conf.bak_pre_remove_allow-query_and_allow-transfer  PROTO.localhost.rev
named.conf.before.mass.import.2003-10-23                  PROTO.localhost-v6.rev
named.conf.orig                                           rndc.conf
named.conf.ORIGINAL                                       s
named.core                                                sendzones.pl
named_head

and all the zonefiles are inside the 's' directory as they are on ns1c. So to restore a zonefile you'd simply copy it from the 's' directory on backup1 back to /etc/namedb/s on ns1c:

scp 69.55.230.11:/data/www/daily/12-12-28/mnt/data1/ns1c-dir/etc/namedb/s/zonefile /etc/namedb/s

For good measure, increment the serial and restart bind.

Similarly, if you need to restore zones deleted from named.conf you can edit named.conf on backup1 and copy the domain blocks over to the live named.conf on ns1c, or you could copy over named.conf itself back to ns1c. This can be dangerous especially if you're potentially overwriting any changes which occurred after the backup was run.

Setup BIND

Below follows steps taken to setup the currently-running bind installs on ns1c/ns2c/ns3c:

cd /tmp
fetch http://www.openssl.org/source/openssl-0.9.8h.tar.gz
tar xzf openssl-0.9.8h.tar.gz
cd openssl-0.9.8h
./config --prefix=/usr --openssldir=/usr/local/openssl
make install clean
./config --prefix=/usr/local --openssldir=/usr/local/openssl
make install clean

(for some reason bind didn’t find all files when installed to /usr so installed again to /usr/local- installed to /usr too to overwrite existing openssl)

cd /tmp
fetch http://ftp.isc.org/isc/bind9/9.5.0-P1/bind-9.5.0-P1.tar.gz
tar xzf bind-9.5.0-P1.tar.gz
cd /tmp/bind-9.5.0-P1
./configure --with-openssl=/usr/local --sysconfdir=/etc/namedb --bindir=/usr/bin --sbindir=/usr/sbin --enable-threads
make install

(copy over files from namedb)

cd /etc/namedb
rehash
rndc-confgen

cat > rndc.conf
# Start of rndc.conf
key "rndc-key" {
        algorithm hmac-md5;
        secret "TywD8Rmk42q+QUbP299C3Q==";
};

options {
        default-key "rndc-key";
        default-server 69.55.230.3;
        default-port 953;
};
# End of rndc.conf

(paste into named.conf)


key "rndc-key" {
      algorithm hmac-md5;
      secret "TywD8Rmk42q+QUbP299C3Q==";
};

controls {
      inet 69.55.230.3 port 953
              allow { 69.55.230.3; } keys { "rndc-key"; };
};

Update 9-17-10

cd /tmp
fetch http://www.openssl.org/source/openssl-0.9.8o.tar.gz
tar xzf openssl-0.9.8o.tar.gz
cd openssl-0.9.8o
./config --prefix=/usr --openssldir=/usr/local/openssl
make install clean
./config --prefix=/usr/local --openssldir=/usr/local/openssl
make install clean

(for some reason bind didn’t find all files when installed to /usr so installed again to /usr/local- installed to /usr too to overwrite existing openssl)

Update perl to 5.8 (5.6 needed)

cd /usr/ports/lang/perl5
make && make install && make clean
rehash
use.perl port

cd /tmp
fetch http://ftp.isc.org/isc/bind9/9.7.1-P2/bind-9.7.1-P2.tar.gz
tar xzf bind-9.7.1-P2.tar.gz
cd /tmp/bind-9.7.1-P2
./configure --with-openssl=/usr/local --sysconfdir=/etc/namedb --bindir=/usr/bin --sbindir=/usr/sbin --enable-threads
make && make install 

rndc stop; bind

add to config:

logging {

// Channels default_syslog, default_debug, default_stderr, and null
// should be predefined, but adjust default_debug here

        channel default_debug {
                file "/var/log/named.debug";
                severity dynamic;
                print-time yes;
                print-category yes;
                print-severity no;
        };

        category default { default_syslog; };
        category general { default_syslog; };
        category database { default_debug; };
        category security { default_syslog; };
        category config { default_syslog; };
        category resolver { null; };
        category xfer-in { default_syslog; };
        category xfer-out { default_syslog; };
        category notify { default_debug; };
        category client { default_debug; };
        category unmatched { default_debug; };
        category network { default_debug; };
        category update { default_syslog; };
        category update-security { default_syslog; };
        category queries { null; };
        category query-errors { default_debug; };
        category dispatch { default_debug; };
        category dnssec { default_debug; };
        category lame-servers { null; };
        category delegation-only { default_debug; };
        category edns-disabled { null; };
};

Setup - ns3c

cd /mnt/data1/
mkdir ns3c-dir
cd /usr/src/
make world DESTDIR=/mnt/data1/ns3c-dir ; pagedave
cd etc
make distribution DESTDIR=/mnt/data1/ns3c-dir

scp backup2:/mnt/data4/bin/freebsd6.x/jailkill /usr/local/sbin
scp backup2:/mnt/data4/bin/freebsd6.x/jailps /usr/local/sbin

scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc/devfs.rules
cat >> /etc/rc.conf
devfs_system_ruleset="devfsrules_show_all"

/etc/rc.d/devfs start
devfs rule showsets
devfs rule -s 3 show
mount -t devfs devfs /mnt/data1/ns3c-dir/dev/
ls /mnt/data1/ns3c-dir/dev/
devfs -m /mnt/data1/ns3c-dir//dev rule -s 3 applyset

cd /mnt/data1/ns3c-dir/
ln -sf dev/null kernel
jail /mnt/data1/ns3c-dir testhostname 192.168.11.100 /bin/sh

csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="ns3c.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver nameserver 66.181.0.2\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username   : user
Password   : <random>
Full Name  : user
Uid        : 1001
Class      :
Groups     : user
Home       : /home/user
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

tzsetup

newaliases 

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit               /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages 

passwd root

exit
exit

#cd libexec
#chflags noschg ld-elf32.so.1
#chflags noschg ld-elf.so.1

#mv ld-elf32.so.1 ld-elf32.so.1-orig
#ln ld-elf.so.1 ld-elf32.so.1

#chflags schg ld-elf.so.1
#chflags schg ld-elf32.so.1


cd /usr/ports
make -DNOCLEANDEPENDS clean 
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/ns3c-dir/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cat >> /usr/local/etc/rc.d/ns3c.sh
mount -t devfs devfs /mnt/data1/ns3c-dir/dev/
devfs -m /mnt/data1/ns3c-dir//dev rule -s 3 applyset
jail /mnt/data1/ns3c-dir ns3c.johncompanies.com 69.55.229.3 /bin/sh /etc/rc

add to .cshrc
set prompt = "`/bin/hostname -s` %/# "

cd /usr/ports/lang/perl5.8
make install clean

cd /tmp
fetch http://www.openssl.org/source/openssl-0.9.8o.tar.gz
tar xzf openssl-0.9.8o.tar.gz
cd openssl-0.9.8o
./config --prefix=/usr --openssldir=/usr/local/openssl
make install clean
./config --prefix=/usr/local --openssldir=/usr/local/openssl
make install clean

(for some reason bind didn’t find all files when installed to /usr so installed again to /usr/local- installed to /usr too to overwrite
existing openssl)

cd /tmp
fetch http://ftp.isc.org/isc/bind9/cur/9.7/bind-9.7.1-P2.tar.gz
tar xzf bind-9.7.1-P2.tar.gz
cd /tmp/bind-9.7.1-P2
./configure --with-openssl=/usr/local --sysconfdir=/etc/namedb --bindir=/usr/bin --sbindir=/usr/sbin --enable-threads
make install

(copy over files from namedb)

cd /etc/namedb
rehash
rndc-confgen

cat > rndc.conf
# Start of rndc.conf
key "rndc-key" {
        algorithm hmac-md5;
        secret "TywD8Rmk42q+QUbP299C3Q==";
};

options {
        default-key "rndc-key";
        default-server 69.55.229.3;
        default-port 953;
};
# End of rndc.conf

(paste into named.conf)


key "rndc-key" {
      algorithm hmac-md5;
      secret "TywD8Rmk42q+QUbP299C3Q==";
};

controls {
      inet 69.55.229.3 port 953
              allow { 69.55.229.3; } keys { "rndc-key"; };
};


adduser ns1
From ns1:
cat /root/.ssh/id_dsa.pub | ssh ns1@ns3c.johncompanies.com 'cat - >> /usr/home/ns1/.ssh/authorized_keys' 

copy over createzone code. on ns2c:
scp /etc/namedb/createzones.pl user@ns3c.johncompanies.com:~
scp /etc/namedb/named_head user@ns3c.johncompanies.com:~

on ns3c:
mv ~user/named_head /etc/namedb/named_head
mv ~user/createzones.pl /etc/namedb/createzones.pl
chown root:wheel /etc/namedb/named_head
chown root:wheel /etc/namedb/createzones.pl
chmod o-rwx /etc/namedb/named_head
chmod o-rwx /etc/namedb/createzones.pl
vi /etc/namedb/createzones.pl
:%s/ns2c/ns3c/g

vi /etc/namedb/named_head
replace RNDC key and set IP

mkdir /etc/namedb/s

cd /usr/ports/mail/p5-Mail-Sendmail
make install clean

crontab -e 
10 12,0 * * * perl /etc/namedb/createzones.pl

update named.conf, /etc/namedb/sendzones.pl on ns1c

on firewall:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984
ipfw add 00096 allow ip from 69.55.229.3 to 69.55.230.2