DNS
DNS Operations
Dns requests make up about 1/3 - 1/2 of all support requests. On the one hand, they are very easy requests to fulfill - but on the other hand, it is important to understand exactly how dns works at johncompanies, and how to repair or rebuild it. Dns is one of two things (the firewall being the other) that could destroy the business forever if it fails and is not solved very quickly.
JohnCompanies runs three dns servers:
ns1c.johncompanies.com 69.55.225.225 ns2c.johncompanies.com 69.55.230.3 ns3c.johncompanies.com 69.55.229.3
All of these are jailed FreeBSD systems - just like any other FreeBSD server instance we would sell to a customer. This means neither no server is its own purpose-built dedicated physical machine.
ns1c is a FreeBSD jail that lives on mail.johncompanies.com ns2c is a FreeBSD jail that lives on jail18 ns3c is a FreeBSD jail that lives on nat2 at i2b
All nameservers were built exactly according to the sections below. A notable reference:
http://www.kozubik.com/docs/bind9_freebsd4.txt
A simple FreeBSD installation was used, then named.conf was dropped into /etc/namedb, and all of the zone files were dumped into /etc/namedb/s
This means that reconstruction due to a failure should be very easy. Since ns1c and ns2c are both backed up nightly, all that needs to be done is a new jailed system created and assigned the proper nameserver IP, bind loaded according to that document, and then the named.conf and the zone files taken from the most recent backup.
It should be noted that ns2c & ns3c have no primary records on it - all they do is slave zones from ns1c so in the case of a ns2c/ns3c failure, the only file you need is named.conf from ns1c. Ns2c & ns3c are automatically updated with zones from ns1c 2x a day on the 12’s (cron runs usr/local/sbin/sendzones.pl on ns1c and a few minutes later /etc/namedb/createzones.pl runs on ns2c & ns3c) so you should never have to touch it. If the replication process fails, an email will be sent to support- the problem is almost always on ns1c.
We interface with these three nameservers as if they were totally seperate, standalone servers. That is to say, when using and administering them, no concession at all is given to the fact that they are actually jailed systems running on some other underlying machine. You ssh to them in the screen window mentioned above just like any other system.
The configuration is very, very simple. Nothing is running on these systems except for bind - nothing else at all. The configuration file is /etc/namedb/named.conf, and consists simply of zone records that look like:
// col00005 zone "burnscustom.com" {
type master;
file "/etc/namedb/s/burnscustom.com";
};
in the case of a master entry, or:
// col00023 zone "spews.net" {
type slave; masters { 198.78.66.18; };
file "/etc/namedb/s/spews.net";
};
in the case of a slave entry. Very very simple. Note that both entries are exactly 6 lines of data with one blank line after/before it before the next zone entry- there shouldn’t ever be any more than 6 lines per definition.
The comment field at the beginning of every zone record in /etc/namedb/named.conf is the customer number. Additional/new zones should be added to the end of a customer’s current zone list.
In the case of slave zones, no other action needs to be taken other than entering the zone into named.conf, and HUPping the nameserver - the zone file will automatically be created in the /etc/namedb/s directory.
In the case of a master record, a new file needs to be created to match the filename in the record. By default, all new records should look like this:
$TTL 43200 @ IN SOA ns1c.johncompanies.com. dns.johncompanies.com. (
2005062302
3600
900
3600000
3600 )
NS ns1c.johncompanies.com.
NS ns2c.johncompanies.com.
MX 10 mail.echosoft.com.
echosoft.com. A 69.55.226.59
- .echosoft.com. A 69.55.226.59
Notice how, by default, there is simply a wildcard entry for the domain ? Almost all of our domains are like that, and we do that by default. Another thing we do by default is make the primary MX record be mail.thatdomain.com.
This requires more explaining. If a new customer comes to us with a domain to put into the dns server, the zone needs to be entered into named.conf as shown above. Then, a new file like the one shown above needs to be created, and their domain name put into it - one line for the domain, and a second line for the wildcard entry. Then, we set the primary MX record to mail.thatdomain.com.
Obviously all of this only applies to master domains, where we do the primary dns for the customer - if we slave the domain from the customer, we have no input in this, and we simply suck over whatever they give to us.
Tools we use that make DNS management easier:
When adding any number of new primary domains, there is an interactive script to make this process easier:
createmaster [customerID] [ip]
Optional arguments to it are the customerID and the IP the domains will resolve to. If not given, the script will ask for them. The script will ask for each domain to be added, one domain per line. To finish, just enter a blank line. It will ask for confirmation that the domains read in are all correct and if confirmed, it will create zone files (in the format listed above, Note: if a zone file already exists, it will notify you and give you the option to overwrite or abort writing that zone file.) and echo to the screen the zone definitions to be added to named.conf. Those definitions should be copied and pasted into the conf file (if the customer has existing zones, the new ones should be added at the end of the customer’s existing zones). Once the new zones are in the conf, issue
rr
which will tell bind to reload zones (rr is a wrapper for rndc reload). You should test to see if the new zones are working properly: ns1c# host zigdon.org 69.55.225.225 If the customer has asked for special entries (CNAME, MX, etc) you’ll have to manually edit the zone files. By default, they will look like above with MX set to mail.domain.tld
When adding new slave domains, there is also an script to make this process easier:
createslave [customerID] [ip]
Optional arguments to it are the customerID and the IP the domains will slave from. If not given, the script will ask for them. The script will ask for each domain to be added, one domain per line. To finish, just enter a blank line. It will ask for confirmation that the domains read in are all correct and if confirmed, it will (in the format listed above) and echo to the screen the zone definitions to be added to named.conf. Those definitions should be copied and pasted into the conf file (if the customer has existing zones, the new ones should be added at the end of the customer’s existing zones). Once the new zones are in the conf, issue rr which will tell bind to reload zones and ask their dns server for the zone info. You should test to see if the new zones info was received by looking at the contents of the new zone file: ns1c# cat zigdon.org If the the file is not found, tell the customer to HUP their dns server.
If a customer requests a change to an existing zone, make the changes to the zone file, and increment the serial by 1 digit. In the example above, the serial is 2001121102. After you’ve saved the file, issue
rr <domain>
That will reload the zone with the new info AND send an update to ns2c. If you forget to increment the serial, the update will not make it to ns2c and that will cause mixed info to be fed when zone lookups for that domain are performed. In that case just increment and rr the domain.
listdomains <customerID>: will list domains belonging to a customerID (per what’s contained in the conf)
1. If it is not already clear, the entries in named.conf are in number order of their customer number. So when adding a new entry, search for the customer number first - if there are already entries for that customer, then put it at the end of those entries. If there are not, make sure it gets placed in named.conf in order, according to the customer number.
2. when you run rr – which tail’s /var/log/messages as a sanity check to make sure nothing untoward got loaded into the config, and that it really did get reloaded. Seeing a lot of failed slavings is normal. Seeing an error related to the zone you just entered is what you’re looking for.
3. If bind ever dies - just `ps auxw` and make sure no named process is running, and then just run:
/usr/sbin/named
Since ns1c/ns2c are monitored by castle access, you will be notified immediately if this happens, and if it happens at non-peak hours, the two minutes total that it should be down will barely be noticed.
4. We must do RDNS for our customers. We set it up for them, when they ask - we do not delegate our ips
IMPORTANT: Be very careful with ns1c - it is the lifeblood of the entire business, and a loss of service on it is one of the few things that could permanently destroy the business. Think twice before running any script that could possibly overwrite any files in /etc/namedb/s, and double check all command lines that you run on this server.
If you’re ever lost, go to isc.org and subscribe to bind9-workers
To update RDNS, edit the appropriate file:
ns1c# vi 69.55.234.rev
Increment the serial by 1, then reload as follows:
ns1c# rr 234.55.69.in-addr.arpa
There are a couple other very helpful tools:
vr [domain]
will open the zone file for editing in vi and after exiting reload the zone. Equivalent to: vi [domain]; rr [domain]
listmasters
this will list all the zones for which we’re acting as a master, and their ttl’s
changettl <ttl>
this will change the ttl’s for all master zones. If given an argument it will set the ttl to that argument. In absence, it will use 43200 (12hrs). It will prompt before changing each zone.
ptr <ip>
this will open up the appropriate zonefile for editing an IP’s PTR record. After saving the file, it will reload the zone.
Setup BIND
Below follows steps taken to setup the currently-running bind installs on ns1c/ns2c/ns3c:
cd /tmp fetch http://www.openssl.org/source/openssl-0.9.8h.tar.gz tar xzf openssl-0.9.8h.tar.gz cd openssl-0.9.8h ./config --prefix=/usr --openssldir=/usr/local/openssl make install clean ./config --prefix=/usr/local --openssldir=/usr/local/openssl make install clean
(for some reason bind didn’t find all files when installed to /usr so installed again to /usr/local- installed to /usr too to overwrite existing openssl)
cd /tmp fetch http://ftp.isc.org/isc/bind9/9.5.0-P1/bind-9.5.0-P1.tar.gz tar xzf bind-9.5.0-P1.tar.gz cd /tmp/bind-9.5.0-P1 ./configure --with-openssl=/usr/local --sysconfdir=/etc/namedb --bindir=/usr/bin --sbindir=/usr/sbin --enable-threads make install
(copy over files from namedb)
cd /etc/namedb
rehash
rndc-confgen
cat > rndc.conf
# Start of rndc.conf
key "rndc-key" {
algorithm hmac-md5;
secret "TywD8Rmk42q+QUbP299C3Q==";
};
options {
default-key "rndc-key";
default-server 69.55.230.3;
default-port 953;
};
# End of rndc.conf
(paste into named.conf)
key "rndc-key" {
algorithm hmac-md5;
secret "TywD8Rmk42q+QUbP299C3Q==";
};
controls {
inet 69.55.230.3 port 953
allow { 69.55.230.3; } keys { "rndc-key"; };
};
Update 9-17-10
cd /tmp fetch http://www.openssl.org/source/openssl-0.9.8o.tar.gz tar xzf openssl-0.9.8o.tar.gz cd openssl-0.9.8o ./config --prefix=/usr --openssldir=/usr/local/openssl make install clean ./config --prefix=/usr/local --openssldir=/usr/local/openssl make install clean
(for some reason bind didn’t find all files when installed to /usr so installed again to /usr/local- installed to /usr too to overwrite existing openssl)
Update perl to 5.8 (5.6 needed)
cd /usr/ports/lang/perl5 make && make install && make clean rehash use.perl port cd /tmp fetch http://ftp.isc.org/isc/bind9/9.7.1-P2/bind-9.7.1-P2.tar.gz tar xzf bind-9.7.1-P2.tar.gz cd /tmp/bind-9.7.1-P2 ./configure --with-openssl=/usr/local --sysconfdir=/etc/namedb --bindir=/usr/bin --sbindir=/usr/sbin --enable-threads make && make install rndc stop; bind
add to config:
logging {
// Channels default_syslog, default_debug, default_stderr, and null
// should be predefined, but adjust default_debug here
channel default_debug {
file "/var/log/named.debug";
severity dynamic;
print-time yes;
print-category yes;
print-severity no;
};
category default { default_syslog; };
category general { default_syslog; };
category database { default_debug; };
category security { default_syslog; };
category config { default_syslog; };
category resolver { null; };
category xfer-in { default_syslog; };
category xfer-out { default_syslog; };
category notify { default_debug; };
category client { default_debug; };
category unmatched { default_debug; };
category network { default_debug; };
category update { default_syslog; };
category update-security { default_syslog; };
category queries { null; };
category query-errors { default_debug; };
category dispatch { default_debug; };
category dnssec { default_debug; };
category lame-servers { null; };
category delegation-only { default_debug; };
category edns-disabled { null; };
};
Setup - ns3c
cd /mnt/data1/
mkdir ns3c-dir
cd /usr/src/
make world DESTDIR=/mnt/data1/ns3c-dir ; pagedave
cd etc
make distribution DESTDIR=/mnt/data1/ns3c-dir
scp backup2:/mnt/data4/bin/freebsd6.x/jailkill /usr/local/sbin
scp backup2:/mnt/data4/bin/freebsd6.x/jailps /usr/local/sbin
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc/devfs.rules
cat >> /etc/rc.conf
devfs_system_ruleset="devfsrules_show_all"
/etc/rc.d/devfs start
devfs rule showsets
devfs rule -s 3 show
mount -t devfs devfs /mnt/data1/ns3c-dir/dev/
ls /mnt/data1/ns3c-dir/dev/
devfs -m /mnt/data1/ns3c-dir//dev rule -s 3 applyset
cd /mnt/data1/ns3c-dir/
ln -sf dev/null kernel
jail /mnt/data1/ns3c-dir testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="ns3c.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf
echo "nameserver nameserver 66.181.0.2\
nameserver 69.55.230.3" >> /etc/resolv.conf
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
rm -rf /etc/periodic/daily/400.status-disks
check /tmp for crap
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
adduser (Add account for user, make sure in wheel group)
Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!
vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM
tzsetup
newaliases
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages
passwd root
exit
exit
#cd libexec
#chflags noschg ld-elf32.so.1
#chflags noschg ld-elf.so.1
#mv ld-elf32.so.1 ld-elf32.so.1-orig
#ln ld-elf.so.1 ld-elf32.so.1
#chflags schg ld-elf.so.1
#chflags schg ld-elf32.so.1
cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/ns3c-dir/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)
rm /mnt/data1/jail-DIR/root/.history
cat >> /usr/local/etc/rc.d/ns3c.sh
mount -t devfs devfs /mnt/data1/ns3c-dir/dev/
devfs -m /mnt/data1/ns3c-dir//dev rule -s 3 applyset
jail /mnt/data1/ns3c-dir ns3c.johncompanies.com 69.55.229.3 /bin/sh /etc/rc
add to .cshrc
set prompt = "`/bin/hostname -s` %/# "
cd /usr/ports/lang/perl5.8
make install clean
cd /tmp
fetch http://www.openssl.org/source/openssl-0.9.8o.tar.gz
tar xzf openssl-0.9.8o.tar.gz
cd openssl-0.9.8o
./config --prefix=/usr --openssldir=/usr/local/openssl
make install clean
./config --prefix=/usr/local --openssldir=/usr/local/openssl
make install clean
(for some reason bind didn’t find all files when installed to /usr so installed again to /usr/local- installed to /usr too to overwrite
existing openssl)
cd /tmp
fetch http://ftp.isc.org/isc/bind9/cur/9.7/bind-9.7.1-P2.tar.gz
tar xzf bind-9.7.1-P2.tar.gz
cd /tmp/bind-9.7.1-P2
./configure --with-openssl=/usr/local --sysconfdir=/etc/namedb --bindir=/usr/bin --sbindir=/usr/sbin --enable-threads
make install
(copy over files from namedb)
cd /etc/namedb
rehash
rndc-confgen
cat > rndc.conf
# Start of rndc.conf
key "rndc-key" {
algorithm hmac-md5;
secret "TywD8Rmk42q+QUbP299C3Q==";
};
options {
default-key "rndc-key";
default-server 69.55.229.3;
default-port 953;
};
# End of rndc.conf
(paste into named.conf)
key "rndc-key" {
algorithm hmac-md5;
secret "TywD8Rmk42q+QUbP299C3Q==";
};
controls {
inet 69.55.229.3 port 953
allow { 69.55.229.3; } keys { "rndc-key"; };
};
adduser ns1
From ns1:
cat /root/.ssh/id_dsa.pub | ssh ns1@ns3c.johncompanies.com 'cat - >> /usr/home/ns1/.ssh/authorized_keys'
copy over createzone code. on ns2c:
scp /etc/namedb/createzones.pl user@ns3c.johncompanies.com:~
scp /etc/namedb/named_head user@ns3c.johncompanies.com:~
on ns3c:
mv ~user/named_head /etc/namedb/named_head
mv ~user/createzones.pl /etc/namedb/createzones.pl
chown root:wheel /etc/namedb/named_head
chown root:wheel /etc/namedb/createzones.pl
chmod o-rwx /etc/namedb/named_head
chmod o-rwx /etc/namedb/createzones.pl
vi /etc/namedb/createzones.pl
:%s/ns2c/ns3c/g
vi /etc/namedb/named_head
replace RNDC key and set IP
mkdir /etc/namedb/s
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean
crontab -e
10 12,0 * * * perl /etc/namedb/createzones.pl
update named.conf, /etc/namedb/sendzones.pl on ns1c
on firewall:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984
ipfw add 00096 allow ip from 69.55.229.3 to 69.55.230.2