= NetHere =

URL: http://nethere.com

<br>
Access to servers is restricted to admin-1.nethere.net. Access to admin-1.nethere.net is restricted to our office and nat.johncompanies.com.

<br>
JCI Hosting acquired the web-hosting customers of NetHere. (Previously, NetHere had aquired customers from Simply Internet, and Z-net).
Customer billing/info are being imported from NetHere to JCI database
URL: https://secure.johncompanies.com/mgmt/index.html

<br>
Nethere has several tools for management. These are restricted to only allow
access from the office, or a few admin's homes.

cacti [http://cacti.nethere.net]
nagios[http://nagios.nethere.net]
toolbox [http://toolbox.nethere.net]
helpdesk [http://helpdesk.nethere.net]
support FAQ [http://www.nethere.net/website/support]
knowledge base [http://support.nethere.net/kb]
phpmyadmin [https://phpmyadmin.nethere.net]

== Virtual Hosts ==

<pre>
66.63.129.106
admin-1.nethere.net
admin-2.nethere.net
pike.nethere.net- cacti, nagios- off
shark.nethere.net- ldap
ntdb-1.nethere.net

66.63.129.105
home-1.nethere.net
ntweb-11.nethere.net
ntweb-6.nethere.net
web-3.schedulecafe.com

66.63.129.104
eel.nethere.net- ldap
mta-1.mail.nethere.net
nscache-1.nethere.net
nsrbl-1.nethere.net
phoenix.nethere.net (ns-1.nethere.net)
relay-1.mail.nethere.net

66.63.129.103
andromeda.nethere.net (ns-2.nethere.net)
nscache-2.nethere.net
nsrbl-2.nethere.net
mailbox-2.nethere.net

66.63.129.102 (vm2.eng.nethere.net)
koi.nethere.net- off- realserver
prov-1.nethere.net- cp.nethere.net, cp.znet.net, cp.simplyweb.net
scribe.nethere.net- prov.nethere.net, scribe.nethere.net
tetra.nethere.net- nethere.com, znet.com, simplyweb.com
web-2.inboxhq.net- helpdesk
webmail-1.nethere.net

66.63.129.101 (c.host.nethere.net)
ahi.nethere.net- ldap
mta-2.mail.nethere.net
relay-2.mail.nethere.net
unixweb-8.nethere.net

66.63.129.91
lists-1.nethere.net
sb-2.nethere.net
mx-1.nethere.net- spooling server
mailx-1.nethere.net- web server outbound mail server

207.167.93.106- vmware6.eng
ntweb-2.nethere.net
unixweb-12.nethere.net (no web sites on server)
svn1.eng.nethere.net- off

207.167.93.108- vmware8.eng
mailx-2.nethere.net
mx-2.nethere.net

207.167.93.110- vmware10.eng
unixweb-11.nethere.net

66.63.129.2 - New HP DL360
fpweb-2.nethere.net
unixweb-6.nethere.net
mta-db.mail.nethere.net

66.63.129.4- vmware9.eng
mysqldb-1.webhost.nethere.net
</pre>

== Provisioning ==

=== Webservers ===

OS: FreeBSD 4.11-RELEASE-p10 #23

==== General ====

<pre>
unixweb-1 through 8

All have the same configuration except for the following exceptions:

unixweb-4: ZN Front Page server (depreciated)
unixweb-5: ZN Front Page, Miva Merchant server
unixweb-6: SI shared counter server (for SI sites): /www/lucy.inetworld.net/htdocs/cgi-bin/Count.cgi

Server Software Installed

Apache
# httpd -v
Server version: Apache/2.0.55
Server built: Apr 5 2006 17:04:01
PHP
# /usr/local/bin/php4 -v
PHP 4.4.2 (cgi-fcgi) (built: Mar 2 2006 09:31:57)

Miva Merchant (unixweb-5)
MySQL
# mysqladmin version -u root -p
Server version 4.0.25

Apache configurations are in the following directories:

/nethere/conf/apache/conf/nvhosts (name based hosting); naming convention is www.domain.tld
/nethere/conf/apache/conf/vhosts (ip based,SSL); naming convention is www.domain.tld (IP information) and www.domain.tld.common (general site information)

NetHere specific scripts are located in:
/nethere/sbin

Checking Server/site status via mod_status apache module:

# apachectl stop
# /usr/local/etc/rc.d/httpd.sh start-status

URL: http://unixweb-#/status (replace # with the unixeweb number)
** NOTE: After viewing status, do the following, otherwise other users can also view status by using .htaccess.
# apachectl stop
# apachectl startssl
</pre>

==== Provisioning ====
Provisioning new sites:

*** All website provisioning is done via NH script (webadd) on sawfish (admin-1.nethere.net) ***
# /nethere/sbin/webadd -h for usage

Note: We've discontinued new website provisioning on all servers except for unixweb-6, unless otherwise noted for domain

==== FTP Password Adjustment ====

===== Unix Server =====
To change an FTP password:

On the customers server as root do

passwd <username>

(The password prompt will not echo entry)

===== Windows Server =====

To change an FTP password:

<pre>
Find the users login (ie aa5014) in the provisioning tool.
Use Remote Desktop to get into the server.
On Desktop find FTP application Icon.
Use that tool to update the user's password.
</pre>

==== Domain Aliasing ====

For domain aliasing:

1) Edit the /named/named.master on phoenix (ns1.nethere.net)
a) Checkout the file; open with vi
# co -l named.master
# vi named.master
b) find the domain that's serving as the master domain, then, following the general format of the file to add the domain aliases under the master domain entry, using the master domain zone file
i.e.
--cut--
zone "domain.tld" { type master; file "master/domain.tld"; };
zone "aliased_domain.tld" { type master; file "master/master_domain.tld"; };
--cut--
c) save the file, then check-in:
# ci -u named.master < /dev/null
d) update the zone records, using the makefile in /named; check for errors
# make new-zone
# tail /var/log/named
**Note: some domains are IP based virtually hosted (i.e. SSL certificate). This requires a modification to the standard aliasing procedure. If this is the case, do the following
e) Create a new zone file called "domain.tld-alias" in /named/master, copy current domain.tld zone file to the domain.tld-alias file, and adjust the "website" records, removing whatever IP based information is there, and replacing with the appropriate $INCLUDE name-based host template.
e.g.
tropicalshade.net:
--cut--
;; ntweb-4
@ IN A 66.63.136.4
www IN A 66.63.136.4
--cut--
tropicalshade.net-alias
--cut--
;; website
$INCLUDE master/nvhost.ntweb-4
--cut--
f) Adjust the named.master zone file in /named accordingly:
i.e.
--cut--
zone "aliased_domain.tld" { type master; file "master/master_domain.tld-alias"; };
--cut--

2) Edit the Apache configuration for the domain on the web server the master domain is hosted on
a) Script is /nethere/sbin/webalias
# /nethere/sbin/webalias -h (for usage)
e.g. # /nethere/sbin/webalias -d www.originaldomain.tld -a www.domainalias.tld

==== CGI Scripting ====

For security, we've implemented suexec on all sites.
All CGI *must* be placed in the cgi-bin/ (and/or cgibin/ directory for unixweb-4, 5) directory

CGI permissions break down as follows:

Ownership: user=username, group=webuser
Permission: cgi-bin/ = 755; files = 755

suexec errors can be found here:
/www/default/logs/suexec_log

cgi errors for a domain can be found in the main apache error logs for the domain:
/www/www.domain.tld/logs/error_log

More information on CGI scripting can be found here:
http://httpd.apache.org/docs/1.3/howto/cgi.html

==== Formmail Provisioning ====

Formmail is provisioned via NH script (newformmail) on server domain is hosted on.
# /nethere/sbin/newformmail -h for usage

Usage: newformmail [-h] [-d domain] [-i ip_address] [-e "email1 email2"]

==== Disabling a site ====

To disable/enable a website, there is a script (webstatus) located on each webserver:
/nethere/sbin/webstatus -h for usage

1) Disable:
# /nethere/sbin/webstatus -r -d -s www.domain.tld

2) Enable:
# /nethere/sbin/webstatus -r -e -s www.domain.tld

==== Stats Provisioning ====

We utilize webalizer for statistics.

Stats are provisioned via NH scripts on server domain is hosted on.

1) Provision stats for the domain (webaddstats_unix):
# /nethere/sbin/webaddstats_unix -d www.domain.tld

2) Run stats for the domain (runstats_unix):
# /nethere/sbin/runstats_unix www.domain.tld

More information on Webalizer can be found here:
http://www.mrunix.net/webalizer
README: ftp://ftp.mrunix.net/pub/webalizer/README

==== Removing a site ====

1) Site removal is primarily done via a NH script (webdel) on the webserver the domain is hosted on:
# /nethere/sbin/webdel -h for usage
# /nethere/sbin/webdel -r -d www.tldomain.com

2) Update DNS accordingly (ns1.nethere.net)
a) Checkout /named/named.master
# co -l named.master
b) Remove line containing domain
c) Checkin /named/named.master
# ci -u named.master < /dev/null
d) Update zones
# make new-zone
e) Move zone files from /named/master to /named/archive
# mv domain

==== starter removal ====

All starter sites on home-1.nethere.net

1) Removal primarily done via a NH script (webdel_home) on home-1
# nethere/sbin/webdel_home -h for usage
i.e. webdel_home -p zn -s startername <== for removing znet

==== Disk Quota increases ====

Quotas are increased by using the "edquota" command:
# edquota username
Note that quotas are in KB, so a conversion is necessary. Also note that the "soft" quota is 1 MB less than the "hard" quota.

The calculations are as follows:
"soft" = ( quota(in MB) - 1 ) x 1024
"hard" = quota(in MB) x 1024
e.g. for a 500 MB quota:
soft => (500 - 1) x 1024 = 510976
hard => 500 x 1024 = 512000

To check a quota:
# quota -v username

You can also use the prototype users:

# edquota -p quota100 username (100 MB)
# edquota -p quota500 username (500 MB)
# edquota -p quota1000 username (1000 MB)

==== Granting of shell - Enterprise packages ====

Shell is granted by the "chsh" command. By default, C shell (csh) is used:
# chsh -s shell_needed username
e.g. # chsh -s csh username

*Note: customers must provide us with the static IP address(es) they will be connecting from. The IP address(es) must be added to the configuration on admin-1.
1) Modify the corresponding rules files on admin-1:
/dist/files/nhweb/etc/ipf.rules.fxp0 , .em0 (RCS controlled)
2) Push updates to servers
# cd /dist/rdist
# gmake nhweb
# gmake nhweb-update
3) Reload the rules on the corresponding server that needs access to
# ipf -Fa -f /etc/ipf.rules

==== SSL certificate installs ====

1) Whois the site for information on Registrant, email - to be used for SSL generation
# whois domain.tld

2) Create SSL self signed certificate, get CSR for customer to sign
a) SSL generation is done by script: /nethere/conf/apache/pki/newsslcert.sh
b) Copy CSR for customer

3) Create/update Apache configuration files via NH script (webadd_ssl)
# /nethere/sbin/webadd_ssl -h for usage
a) webadd_ssl [-h] [-d domain] [-s ssl_domain] [-n]
# webadd_ssl -d www.domain.tld -s www.domain.tld

4) Update DNS zone with new IP address - done on ns1.nethere.net
a) Check out DNS zone file in /named/master
# co -l domain.tld
b) Adjust A records:
--cut--
;; unixweb-## << enter the unixweb server number for ease of ID
@ IN A vhost_ip_address
www IN A vhost_ip_address
--cut--
c) Check in DNS zone file
# ci -u domain.tld < /dev/null
d) Reload zone file
# rndc reload domain.tld
e) check /var/log/named for errors

5) Update DNS PTR record for IP address - done on ns1.nethere.net
Note: Assuming IP address a.b.c.d
a) Check out in-addr.arpa zone for IP address in /named/master
# co -l a.b.c
b) Follow format for PTR records --cut--
d IN PTR www.domain.tld.
--cut--
c) Check in PTR zone file # ci -u a.b.c < /dev/null
d) Reload PTR zone
# rndc reload c.b.a.in-addr.arpa
e) check /var/log/named for errors

==== Updating SSL certificate ====

1) cd to /nethere/conf/apache/pki/ssl.crt on server site is hosted on

2) Check out (RCS) www.domain.tld.crt file
# co -l www.domain.tld.crt

3) Edit the file, remove old certificate, paste in new certificate

4) Check in (RCS) the www.domain.tld.crt file
# ci -u www.domain.tld.crt < /dev/null

5) Check, restart Apache
# apachectl configtest
# apachectl stop
# apachectl startssl

6) Verify httpd started:
# ps auxw | grep httpd
a) If no processes, will need to revert back to old SSL cert, and restart apache. Check logs for errors
# view /www/default/logs/ssl_engine.log
b) Check for "Unable to configure RSA server private key" and "key values mismatch" entries - this means a bad SSL certificate

==== Password protection ====

Http (simple) password protection is governed by the Apache configuration for the domain

1) Create userdb, users file in the domain root directory (/www/www.domain.tld):
# mkdir userdb
# cd userdb
# htpasswd -bc users username password

2) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>
# co -l www.domain.tld(.common)

3) Edit Apache configuration, add the following lines within the VirtualHost container
--cut--
<Directory "/www/www.domain.tld/dir_to_be_protected">
AuthType Basic
AuthName "www.domain.tld/dir_to_be_protected authentication"
AuthUserFile /www/www.domain.tld/userdb/users
<Limit GET POST>
require valid-user
</Limit>
</Directory>
--cut--

4) Check in Apache config
# ci -u www.domain.tld(.common) < /dev/null

5) Restart Apache
# apachectl configtest
# apachectl restart

More on http (simple) password protection can be found here:
http://httpd.apache.org/docs/1.3/howto/auth.html#basic

==== .htaccess ====

Used if customers want control of certain Apache directives (i.e. Authentication, etc.)

1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>
# co -l www.domain.tld(.common)

2) Add the AllowOverride directive in the <Directory> section, under the PHP FCGIWrapper
i.e.
<Directory "/www/www.domain.tld/htdocs">
FCGIWrapper /www/www.domain.tld/htdocs/cgi-bin/php4 .php
AllowOverride AuthConfig FileInfo Indexes Limit
</Directory>

3) Check in Apache config
# ci -u www.domain.tld(.common) < /dev/null

4) Restart Apache
# apachectl configtest
# apachectl restart

More info on Allow Override can be found here: http://httpd.apache.org/docs/1.3/mod/core.html#allowoverride

==== Domain re-provisioning ====

## NOTE: Following is for domain being provisioned on same server (i.e. just being renamed)

1) Adjust DNS - ns1.nethere.net
a) Rename the DNS zone file to the new domain, remove the old DNS zone files from /named/master and /named/master/RCS
b) Check in the new DNS zone file
c) Edit /named/named.master - replace the old domain with the new one, alias as necessary
d) Reload DNS zones - make new-zone in /named

2) Adjust Apache config - server domain is hosted on
a) Find current config file(s) in /nethere/conf/apache/conf/<vhosts,nvhosts>
b) Replace old domain name entries with new domain name, alias as necessary
c) Save as new_domain.tld
d) Remove old_domain.tld(.common), RCS/old_domain.tld(.common)
e) Check in new_domain.tld via RCS:
# ci -u new_domain.tld < /dev/null
f) Check out Apache include configuration file via RCS /nethere/conf/apache/conf/<nvhosts.conf,vhosts.conf>
g) Edit nvhosts.conf or vhosts.conf, replace old_domain.tld entries with new_domain.tld
h) Check in via RCS /nethere/conf/apche/conf/<nvhosts.conf,vhosts.conf>

3) Rename directory for new domain
# cd /www
# mv www.old_domain.tld www.new_domain.tld

4) Adjust PHP stub files/configuration
a) Adjust php.ini file in /www/www.domain.tld/(php4,php5)
b) Adjust PHP stub files in /www/www.domain.tld/htdocs/cgi-bin/(php4,php5)
*note: need to chflags to "noschg" for /www/www.domain.tld/htdocs/cgi-bin/(php4,php5) before being able to update the stub files
after adjusting, be sure to chflags schg /www/www.domain.tld/htdocs/cgi-bin/(php4,php5)

5) Edit password file (vipw), replace old_domain.tld entries with new_domain.tld
# vipw

6) Restart Apache
# apachectl configtest
# apachectl restart

## NOTE: Following is for domain being re-provisioned on new server

1) Run NH script "webadd" on sawfish to provision domain on new server, *DO NOT* reload DNS, use same user/pass as before.
2) After customer has uploaded site to new server and gives the ok do the following:
a) Update DNS zone for domain.tld; reload zone for domain.tld
b) Wait 48 hours, then remove the site off the old server
# /nethere/sbin/webdel

==== FTP space provisioning ====

Generally FTP sites are provisioned on the server that hosts the main website. In the case of NT based FTP sites, we usually provision them on the server with the most space available.

1) Provision site on server
a) Done via NH script (webadd_ftp): webadd_ftp [-h] [-d domain] [-u username] [-p password] [-n]
# /nethere/sbin/webadd_ftp -h <- for usage
EX: for ftp.domain.tld:
# /nethere/sbin/webadd_ftp -d ftp.domain.tld -u username -p password
(leaving off the -n will restart the proftpd process)
b) Note: The host IP address will be given when the provisioning is completed, use that ip (a.b.c.d) for DNS entries
c) Note: If this is an existing customer on the server, you'll need to increase the quota manually by 100 MB for the customer, see the section on quota increases for more info.

2) Adjust DNS for domain on ns1.nethere.net
a) Checkout zone for domain
b) Add ftp host entry for domain, adjust serial
EX for domain.tld:
--cut--
ftp IN A a.b.c.d
--cut--
c) Check in zone for domain
d) Reload zone
# rndc reload domain.tld
3) Adjust PTR record for domain, adjust serial
a) Checkout zone for a.b.c
b) Add record for domnain:
--cut--
d IN PTR ftp.domain.tld.
--cut--
c) Check in zone for a.b.c
d) Reload zone
# rndc reload c.b.a.in-addr.arpa
4) Check for DNS errors
a) tail /var/log/namedb

==== SiteBuilder provisioning ====

unixweb-7.nethere.net

*Must re-provision site on unixweb-7.nethere.net (if not already done)

1) Log in to SB admin:
http://sitebuilder.nethere.net/admin
a) username: root

2) Add site to SB config
a) Click on Site Management -> Add regular
* Alias is website username: i.e. aa####
* Check the "Active" box
* Plan is "BasePlan"
* Password same as website
b) Click on "Publish Properties
* Check "Allow publishing"
* Site host name: www.domain.tld
* FTP host: unixweb-7.nethere.net
* FTP login/password: same as site user/pass
* FTP working directory: leave blank
c) Click Apply

==== Name Servers ====

OS: FreeBSD

==== General ====

ns1.nethere.net - Primary name servers for DNS zone records
/named - contains the files that have all DNS domain zone entries (named.master, named.slave, named.acl) plus Makefile for distributing DNS records
/named/master - contains all the domain zone files for which we are authorative for, as well as IP address (PTR records)

ns2.nethere.net - Secondary (slave) name server for DNS zone records

nsrbl-1.nethere.net - RBL (Realtime Blackhole List) DNS server
/named/rbldns/cache - contains the files for domains that we specifically allow or deny

nscache-1,2 - caching name servers

==== Adding DNS website entries ====

1) Create a DNS zone file for domain.tld via NH script (zoneadd_vhost)
# /nethere/sbin/zoneadd_vhost -h (for usage)

==== Adding DNS IP entries ====

1) Create the forward and reverse DNS records via NH script (zone_generate) for a netblock, will create /tmp/customer.forward and /tmp/customer.reverse files to be read
a) /nethere/sbin/zone_generate -h for usage
i.e. for netblock a.b.c, starting IP d, ending IP z
# zone_generate -n a.b.c -b d -e z -p customer

2) Checkout, edit the IP in-addr.arpa zone file, reload the zone
a) # co -l a.b.c
b) Seach for the nearest netblock area for the domain, follow format for customer info, read in the /tmp/abbrev.rdns file accordingly, increase Serial for zone in YYYYMMDD## format
e.g.
--cut--
;;;;
;; 66.63.152.232/30 (255.255.255.252)
;; Description: First Choice Home Improvement
;; Contact: Shannon Hill <firstchoicehi@hotmail.com>, (858) 277-5351
;; Location: AR-1, Serial3/0/18:0
;;;;
232 IN PTR firstchoice-net.access.nethere.net.
233 IN PTR firstchoice-gw.access.nethere.net.
234 IN PTR firstchoice-2.access.nethere.net.
235 IN PTR firstchoice-bcast.access.nethere.net.
--cut--
c) # ci -u a.b.c < /dev/null
d) # rndc reload c.b.a.in-addr.arpa
e) verify loading of zone: # tail /var/log/named
3) Checkout, edit the forward DNS zone file, reload the zone
a) # co -l access.nethere.net
b) Search for the nearest neblock area for the domain, follow format for customer, read in the /tmp/abbrev.fdns file accordingly, increase Serial for zone in YYYYMMDD## format
e.g.
--cut--
;; 66.63.152.232/30 (255.255.255.252)
firstchoice-net IN A 66.63.152.232
firstchoice-gw IN A 66.63.152.233
firstchoice-2 IN A 66.63.152.234
firstchoice-bcast IN A 66.63.152.235
--cut--
c) # ci -u access.nethere.net < /dev/null
d) # rndc reload access.nethere.net
e) verify loading of zone: # tail /var/log/named

==== Unblocking RBLd IP addresses ====

There are two primary reasons why we add customers to the allow relay list:
a) They have a static IP address (i.e. DSL, T1) and wish to use our mail servers
b) They've been blocked by one of our subscribed blackhole lists, however, have patched their machine and are no longer open to relay.

To do this on nsrbl-1.nethere.net:

1) Checkout the allow.relays.nethere.net file located in /named/rbldns/cache

2) Edit the file, and add the IP address in the following format:
--cut--
a.b.c.d YYYYMMDD hostname reason for listing
--cut--

You can also add subnets via '/' notation for relay
--cut--
a.b.c.d/28 YYYYMMDD hostname reason for listing
--cut--

3) Check in the file

4) Updates to the rbldns zone are done automatically on the hour, every hour, so no need to do anything else.

Note:

For "permanent" (i.e. customer static IP addresses), add the IP address in the "## permanent allowed relay (i.e. customer w/ static IP)" section
For "temporary" (i.e. blackhole listed IP addresses), add the IP address in the "## temporary" section

Also, for temporary IP addresses, we need to send the note to the requester detailing the following:
a) If the IP address relays Spam/UCE/Viruses through us, it is to be removed permanently.
b) The customer must follow the steps listed on the blackhole list the IP was listed on to get removed.

==== Flushing DNS cache for a domain ====

Due usually to a bad zone or excessively long TTL for a domain, the cache for it will need to be flushed.

On nscache-1:
# /usr/local/sbin/rndc flushname domain.tld

If cache is still corrupted (i.e. zone lookups either fail or are incorrect), need to stop and restart the caching server:

# /etc/init.d/local.named stop
# /etc/init.d/local.named start

==== SiteBuilder ====
http://sitebuilder.nethere.net/
-URL used to test sitebuilder

http://sitebuilder.nethere.net/Login
-URL for control panel.
<p>This is where the customer also logs in to manage
their web site and also where you log in to administer sitebuilder.
Sitebuilder is hosted on sb-2.nethere.net. The site is designed/built on
this server and published to unixweb-7.nethere.net. This is the only server
that can host a sitebuilder web site. The admin log in to manage
sitebuilder is:

Username: admin
Password: N3tH3r31!

==== Cart32 ====
cart32 information:

Cart32 is hosted on ntweb-6.nethere.net. That is where all of the
configuration files are located. The location of the ini file is
D:\websites\Cart32cgi/cart32.ini. This is where ip restrictions to admin
panel are set, password can be reset for admin, time limit restriction
reset, etc.

There are four customers that still use cart32:

<pre>
https://www.cart.simplyweb.net/lab400/cart/c32web.exe
https://www.cart.simplyweb.net/nutragenics/cart/c32web.exe
https://www.cart.simplyweb.net/retrogen/cart/c32web.exe
https://www.cart.simplyweb.net/stonesculptorssupplies/cart/c32web.exe
</pre>

The client codes are lab400, nutragenics, retrogen, and
stonesculptorssupplies. You can reset their passwords through the admin
panel.

To administer cart32:

Control Panel: https://www.cart.simplyweb.net/cart/c32web.exe/Admin
Username: administrator
Password: N3tH3r31!
Cart Admin Password: N3tH3r31!

=== Webmail ===

==== Removing webmail filters ====

All webmail filters are stored in the MySQL database on webmail-1.

Simple SQL commands:

* Choosing a DB
mysql> use db_name;

* Showing tables in a DB (must be using a DB)
mysql> show tables;

* Describing a table's fields
mysql> describe table_name;

For NetHere, the database is horde_nh.
For Simply, the database is horde_si.
For zNET, the database is horde_zn.

1) Log to MySQL as root, using the normal root password
a) # mysql -u root -p
2) Select the appropriate database
a) mysql> use horde_nh
3) Delete webmail filters.
a) mysql> delete from horde_prefs where pref_uid='<username>' and pref_name='filters';
Replace <username> with the users login.

==== Removing Address book entries ====

1) Log in
2) Select appropriate db
3) Find object_id, delete
a) mysql> select object_id from turba_objects where object_email='email@domain.tld';
b) mysql> delete from turba_objects where object_id='object_id_obtained_previously';

==== Repairing webmail address book ====

1) Export (via webmail, if possible) the address book to a csv file.

2) Delete the Turba objects in the mySQL database in the turba_objects table that is assigned to the customer.
mysql> delete from turba_objects where owner_id='username';

3) Import the saved address book via webmail.

==== Changing webmail From Information ====

1) Login as the customer (http://webmail.nethere.net)

2) Click on Options

3) Click on Personal Information

4) Select either default identity or a new one and click on Edit Your Identities

=== MySQL ===

All unixweb-## boxes have MySQL processes running on them. DB provisioning is done on the same server as which hosts the domain.
For Windows 2000 MySQL provisioning, all DBs are placed on ntdb-2.nethere.net

==== MySQL DB provisioning ====

Use https://phpmyadmin.nethere.net
User: root
PW: <root db pw>

1) Select the server that the DB needs to be hosted on, typically the same server that the website is hosted on.
2) Create the DB
a) Under "MySQL" - "Create new database" - the DB name is typically the website username; click "Create"
3) Create the User, set permissions for user on DB
a) Click "Home" (upper left), then click "Privileges" (Under "MySQL")
b) Click "Add a new User" make sure "Any host" selected, "User name" is typically the website username, "Password" is typically the website password. !LEAVE "Global Privileges" settings deselected! Click "Go" (bottom right corner) when finished.
c) Under "Database-specific privileges" be sure to add the user database
d) Select DB, then for "Privileges:" make sure the following are checked:
Select, Insert, Update, Delete, Create, Alter, Index, Drop, Create Temporary Tables, Lock Tables, References

=== List Server ===

lists-1.nethere.net

==== Mail List provisioning ====

1. Go to the /mailman/bin directory:
# cd /mailman/bin

2. Create the list on lists-1:
# ./newlist -q \
listname@lists.domain.com \
mailman-owner@lists.domain.com \
password

3. Configure the list using default settings:
# ./config_list -i /mailman/data/defaultlist.cfg listname

*** IF A NEW DOMAIN FOLLOW INSTRUCTIONS BELOW ***

NOTE: All files in /etc/mail are controlled by RCS

4. Go to the /etc/mail directory:
# cd /etc/mail

5. Add the list domain to the mailertable file:
lists.domain.com mailman:lists.domain.com

6. Add the list domain to the relay-domains file.

7. Add the list domain to the virtuserdomain file.

8. Add the following entries to the virtusertable file:
mailman@lists.domain.com <customer_email_address>
mailman-owner@lists.domain.com mailman@lists.domain.com

9. Run make to rebuild all files:
# make

10. Run make to restart sendmail:
# make restart

=== Backup Server ===

backup-1 -> odd numberd unix
backup-2 -> even numbered unix

On Saturdays, we do a level 0 dump
the rest are differential

Restoration fees are basically $25 per day restored for email, $150 for website from cancelled archive, $25 a day for website

Restoration is based on customer request. In order to restore a site/mailbox fully, you must start restoration from the previous level 0 backup and continue until the day after the day requested, since backups are performed in the morning.

General rule is to leave the gzipped file available for 24 hours, after that remove it.

==== Site Restoral ====

If the site was removed by the web_del script, then so long as it is within 30 days since removal, the gzipped site will be available within /www/archive/www.domain.tld-date_removed.tar.gz
Extract with tar:
# tar -zxvpf www.domain.tld-date_removed.tar.gz

If the site needs to be restored from backup, here are the steps

1) To determine where backups are stored, go to /nethere/conf/backup

2) grep for the server within the directory to determine the backup directory used
i.e.
# grep unixweb-1 *

4) Go to /backup1/unix/unixweb-1.nethere.net
a) you'll see a bunch of directories, named <date>-<dump_level>

5) Since the backups happen in the morning, typically you'll need to go to the day before, restore
e.g. for 11/10
a) cd 20041109-3
b) restore -if and whatever the filesystem is named
(for interactive: add files/dirs, extract, then use 1 for volume #)
i.e.
restore> add <path_to_dir/files>
restore> extract
(for volume #: 1)
(Set owner: y)

6) After you've restored the files, just use scp as root on sawfish to copy them off of backup-# and then onto unixweb-#
a) (on sawfish - two step process)
# scp backup-1:/tmp/<restored_file> /tmp
# scp /tmp/<restored_files> unixweb-1:/tmp

7) Cleanup any restored files off of backup-# server and sawfish

With the newly restored files, following are the steps necessary to restore a site:

1) Restore www.domain.tld directory to /www
# mv /www/archive/www.domain.tld /www

2) Restore configuration files
a) Apache: mv /www/archive/nethere/conf/apache/(n)vhosts/www.domain.tld /nethere/conf/apache/conf/(n)vhosts/
b) Webalizer: mv /www/archive/nethere/conf/webalizer/unix/www.domain.tld /nethere/conf/webalizer/unix

3) Add user to /etc/master.passwd:
a) vipw
(go to end of file, read in the master.passwd file from /www/www.domain.tld)
:r /www/www.domain.tld/master.passwd
b) Remove /www/www.domain.tld/master.passwd

4) Change flags on cgi-bin/ directories:
# chflags sunlnk /www/www.domain.tld/htdocs/cgi-bin/
# chflags schg /www/www.domain.tld/htdocs/cgi-bin/php*

5) Add configuration file entry to nvhosts/vhosts.conf
# co -l (n)vhosts.conf
# vi (n)vhosts.conf
# ci -u (n)vhosts.conf

6) Check apache, restart; verify apache
# apachectl configtest
# apachectl graceful
#

7) Cleanup remaining restored files
# rm -r /www/archive/www
# rm -r /www/archive/nethere

8) Enter zone into DNS on ns1.nethere.net
a) zone file
# mv /named/archive/domain.tld /named/master
# mv /named/archive/domain.tld,v /named/master/RCS
b) named.master file
# co -l /named/named.master
(add zone)
# ci -u /named/named.master
# cd /named; make new-zone
# tail /var/log/named (look for errors)
*************************************************
(12:40:47 PM) Henry Chan: restore is now available that is compatible with the 4.4bsd format
(12:41:04 PM) Henry Chan: to restore, use the following command: restore -c -i -f path_to_archive
(12:41:08 PM) Henry Chan: the "-c" is what does it
(12:41:29 PM) Henry Chan: (only applies to backup-1-new... doesn't work on backup-2 or backup-4)

==== Email Restoral ====
<pre>
Storage path prefixes:
/nfs/1 is sndg-netapp-1 (on backup-4 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)
/nfs/2 is sndg-netapp-2 (on backup-3 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)
/nfs/3 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)
/nfs/4 is sndg-netapp-1 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)
/nfs/5 is sndg-netapp-2 (on backup-1 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)
/nfs/6 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)

Looking up a mailbox's Storage Path:
You will need to know which directory and NFS server the customer's mail is
stored on. Use the Provisioning Tool to get the Storage Path (find the customer's mail
account, then click on the Engineering sub-tab under the Email tab). You should end up
with something like "/nfs/3/nh/h/t/htchan/Maildir". Match the prefix of the storage
directory with an NFS server (see prefixes above).

1) If restoring a recently deleted mail account:
a) Look for the archive on mailbox-1 in /nfs/archive/mail/{platform}/{username}.{date}-{PID}.tar.gz. If it doesn't exist, it's been too long and the only way to get email back is to restore from backup.
b) Make sure the account is re-created in Provisioning Tool and look up the storage path (see "Looking up a mailbox's Storage Path" above).
c) Extract the archive to a temporary directory:
Sample command:
cd /tmp; tar xzvpf /nfs/archive/mail/nh/zella.20090902-7508.tar.gz
Sample output:
nfs/2/nh/z/e/zella/
nfs/2/nh/z/e/zella/Maildir/
nfs/2/nh/z/e/zella/Maildir/tmp/
nfs/2/nh/z/e/zella/Maildir/new/
nfs/2/nh/z/e/zella/Maildir/cur/
nfs/2/nh/z/e/zella/Maildir/maildirsize
nfs/2/nh/z/e/zella/Maildir/.Trash/
nfs/2/nh/z/e/zella/Maildir/.Trash/tmp/
nfs/2/nh/z/e/zella/Maildir/.Trash/new/
nfs/2/nh/z/e/zella/Maildir/.Trash/cur/
nfs/2/nh/z/e/zella/Maildir/.Trash/maildirfolder
nfs/2/nh/z/e/zella/Maildir/.Drafts/
nfs/2/nh/z/e/zella/Maildir/.Drafts/tmp/
nfs/2/nh/z/e/zella/Maildir/.Drafts/new/
nfs/2/nh/z/e/zella/Maildir/.Drafts/cur/
nfs/2/nh/z/e/zella/Maildir/.Drafts/maildirfolder
nfs/2/nh/z/e/zella/Maildir/.Sent Items/
nfs/2/nh/z/e/zella/Maildir/.Sent Items/tmp/
nfs/2/nh/z/e/zella/Maildir/.Sent Items/new/
nfs/2/nh/z/e/zella/Maildir/.Sent Items/cur/
nfs/2/nh/z/e/zella/Maildir/.Sent Items/maildirfolder
nfs/2/nh/z/e/zella/Maildir/courierpop3dsizelist
Note:
In the above output, notice that the Maildir is located in "nfs/2/nh/z/e/zella", it will be used in the next step.
d) Copy the Maildir files to the new mailbox storage path using tar (tar does a better job at preserving things like symbolic links and permissions than cp or mv):
Sample command:
cd /tmp/nfs/2/nh/z/e/zella; tar cf - Maildir | (cd /nfs/3/nh/z/e/zella; tar xvpf -)
Output:
You should list the files as it is copied. Basically, similar to what you saw in step "c" above but without the "nfs/2/nh/z/e/zella" prefix.
Note:
The command format is basically:
cd /tmp/{restored-maildir-path-see-note-in-step-c}; tar cf - Maildir | (cd {new-storage-path-without-Maildir}; tar xvpf -)
e) Clean up temporary directory:
Sample command:
cd /tmp; rm -rf nfs

2) If restoring a mailbox to a particular date's backup:
a) Look up the storage path to the customer's mailbox and determine which backup server the dump file is on (see "Looking up a mailbox's Storage Path" and "Storage path prefixes" above).
b) Restore the level 0 and any incremental backups (in order) to /tmp on the backup server:
Sample commands:
cd /tmp
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090912-0/mail.dump
chflags -R 0 /tmp
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090913-1/mail.dump
chflags -R 0 /tmp
...skipped repetitive stuff here...
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090918-6/mail.dump
chflags -R 0 /tmp
Note:
After each restore, we need to recursively remove all flags from /tmp to eliminate the immutable flag that gets set on the files (this happens only on NetApp dumps... who knows why).
c) Create a new tar of the Maildir directory so that it can be copied to mailbox-1 for further processing:
Sample command:
cd /tmp/nh/z/e/zella; tar cf /tmp/archive.tar Maildir
d) Clean up /tmp:
Sample command:
rm -rf /tmp/nh
Note:
Depending on the platform, it might be /tmp/nh, /tmp/si, or /tmp/zn.
e) Copy the /tmp/archive.tar file to /tmp on mailbox-1.
f) On mailbox-1, extract the Maildir archive on top of what they already have:
Sample command:
cd /nfs/2/nh/z/e/zella; tar xvpf /tmp/archive.tar
g) Remove /tmp/archive.tar.
</pre>

==== Beginning of old instructions ====
Use the same techniques as site restoration, with the following exceptions:

1) Since everything is in maildir format, you have to reassmble things in /tmp/<username>, tar it up, and then restore it on any of the mailbox machines
a) using the following syntax:
# tar -zcpf /tmp/username.tgz yyyymmdd-#/platform yyyymmdd-#/platform yyyymmdd-#/platform
i.e.
# tar -zcpf /tmp/username.tgz 20050915-5/nh 20050914-4/nh 20050910-0/nh

2) In regards to mail spools, you'll need to use the chflags command to adjust the flags on the files. For whatever reason, they are stored with a system immutable flag
a) Do chflags -R noschg <dir> on the restored directory before copying the files with sawfish otherwise, the files cannot be deleted:
i.e.
# chflags -R noschg /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/
b) After tar/gzipping, remove the restored directories
i.e
# rm -r /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/nh

3) Can restore quickly with script on mailbox-1 (or mailbox-2) using the NH script "restoremail"
a) # /nethere/sbin/restoremail -h for usage
i.e.
# restoremail -p nh -f nhusername.tgz -n 2 -u nhusername
*** End of old instructions ***

* Backup locations *

backup-1:
backup-3.nethere.net
fpweb-1.nethere.net
home-1.nethere.net
koi.nethere.net
marmaduke.inetworld.net
ntdb-1.nethere.net
ntweb-1.nethere.net
ntweb-3.nethere.net
ntweb-5.nethere.net
ntweb-7.nethere.net
phoenix.nethere.net
unixweb-1.nethere.net
unixweb-3.nethere.net
unixweb-5.nethere.net
unixweb-7.nethere.net
shark.nethere.net
tetra.nethere.net
wms-1.nethere.net

backup-2:
andromeda.nethere.net
backup-4.nethere.net
eel.nethere.net
fpweb-2.nethere.net
ntdb-2.nethere.net
ntweb-2.nethere.net
ntweb-4.nethere.net
ntweb-6.nethere.net
unixweb-2.nethere.net
unixweb-4.nethere.net
unixweb-6.nethere.net
ds.znet.com
mx1.znet.com
mx2.znet.com
mx3.znet.com
dmx.znet.com
la.znet.com
uf.znet.com

backup-3:
backup-1.nethere.net
lists-1.nethere.net
mailbox-1.mail.nethere.net
mailbox-3.mail.nethere.net
mta-1.mail.nethere.net
mx-1.nethere.net
nscache-1.nethere.net
nsrbl-1.nethere.net
pegasus.nethere.net
scan-1.mail.nethere.net
sndg-netapp-2.nethere.net
relay-1.mail.nethere.net
webmail-1.mail.nethere.net

backup-4:
ahi.nethere.net
backup-2.nethere.net
cp-1.nethere.net
cygnus.nethere.net
dragon.nethere.net
lisa.nethere.net
mailbox-2.mail.nethere.net
mta-2.mail.nethere.net
news-1.nethere.net
nscache-2.nethere.net
nsrbl-2.nethere.net
sawfish.nethere.net
scan-2.mail.nethere.net
scribe.nethere.net
sndg-netapp-1.nethere.net
relay-2.mail.nethere.net
urchin.nethere.net

=== Mail servers ===
==== Organization ====
<pre>

NetHere Mail Servers
====================

Customers Inbound Mail Outbound Mail
/\ || || /\
|| || || smtp ||
|| || \/ ||
|| || ServerIron ||
|| || || ||
|| || || ||
|| || smtp \/ ||
|| || =========> mta-1 mta-2 ====> relay-1 <==== unixweb-* servers
|| || || /\ relay-2 ntweb servers
http || pop3 || || ||
|| imap || \/ +--------------+
|| || ServerIron |
|| || || |
|| || || |
|| || \/ |
|| || nsrbl-1 nsrbl-2 |
|| || (spamhaus, dcc blacklist) |
|| || || |
|| \/ \/ |
\/ mailbox-1 <=== scan-1 scan-2 |
webmail <===> mailbox-2 (sendmail/amavisd/sophos) |
imap mailbox-3 |
/\ /\ |
|| || \/
nfs || +---------------------------> mta-db
|| (LDAP)
\/
sndg-netapp-1
sndg-netapp-2
sndg-netapp-3

</pre>

==== Overview ====
mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.
mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.

nsrbl-1.nethere.net- Realtime blacklist check using dccd and spamhaus. The mta servers query the ip address of the mail server against the blacklist before transferring the email to the scanning servers.

nsrbl-2.nethere.net

scan-2.mail.nethere.net- Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.
scan-2.mail.nethere.net Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.

mailbox-1.mail.nethere.net- Front end server for the mail directories. Queries LDAP server for miscellaneous customer information including the directory the email should be stored in, passwords, spam sensitivity levels, etc. before delivering email to appropriate directory.
mailbox-2.mail.nethere.net
mailbox-3.mail.nethere.net

Mail is stored in maildir format:

/nfs/<nfs_number>/<platform>/u/s/username

sndg-netapp-1-new.nethere.net- Solaris system using zfs set up as a nfs mount on mailbox-1, -2, and -3. Email is stored on these platforms for all customers.
sndg-netapp-2-new.nethere.net
sndg-netapp-3-new.nethere.net

mx-1.nethere.net- spooling mail server

mailx-1.nethere.net- all outbound mails originating from a web server is sent out this email server. All web servers are set to use mailx.nethere.net which resolves (through the serveriron) to either mailx-1.nethere.net and mailx-2.nethere.net. Since we shut down mailx-2, all outbound mail from the web servers goes through mailx-1. This was to prevent all the customers email from being blacklisted as a result of a web site being compromised.

mta-db.mail.nethere.net- database server for the mta servers. All postfix databases are stored here.

webmail.nethere.net- Hosts the webmail program for webmail.nethere.net, webmail.simplyweb.net and webmail.znet.net.

==== Client mail settings ====

Inbound Mail Server:

POP3
pop3.nethere.net
pop3.znet.net
pop3.simplyweb.net

IMAP
imap.nethere.net
imap.znet.net
imap.simplyweb.net

Outbound Mail server:

mail.nethere.net
mail.znet.net
mail.simplyweb.net

Webmail:

webmail.nethere.net
webmail.znet.net
webmail.simplyweb.net

Control Panel:

cp.nethere.net
cp.znet.net
cp.simplyweb.net

==== nsrbl Disk Quota Warning ====

In the event you get a nagios regarding space on the two nsrbl servers, it is most likely the result of the dccd databases getting too large. To clear out space, log into each nsrbl server and execute the following:

df -h

cd /var/dcc/libexec

./stop-dccd

cd /var/dcc

rm dcc_db*

sync

sync

cd /var/dcc/libexec

./start-dccd

ps auxw | grep dccd

====Update outbound quota for a single ip address====

We restrict the total number of emails that a customer can send to 2000 email recepients per 24 hour period. This is tracked by the ip address of the computer that connects to the mta server when it sends an outbound emal. The ip address and count is stored in the postfix database server, mta-db.mail.nethere.net. To increase the quota for a single ip address, you can log into either mta-1, mta-2, or console direcly into the database server. To use one of the mta servers:

<pre>
1. Log onto mta-1.mail
2. Switch to mysql
$mysql -h 10.0.0.69 -u policyd_outbound -p
username: policyd_outbound
password: p0stf1x!
3. Review database information
mysql>show databases;
4. Switch to the policyd table.
mysql>use policyd;
5. Review table header information
mysql>show tables;
mysql>describe throttle;
6. Verify the ip address is above quota
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx'; #this is the ip address of the account you are increasing.
7. Increase the maxium number of recepients the ip address can send to
mysql>update throttle SET _rcpt_max = _rcpt_max + 20000 where _from='24.249.205.66';
8. Verify
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx';
</pre>

====Remove a mail server from policyd blacklist====

The ip address of a mail server can be placed on a blacklist for several reasons, to high of rate of inbound email from a single ip address, incorrectly formatted helo, etc. In the event you need to remove a mail server's ip from the policyd blacklist, you can log into either mta server of connect directly to the policyd database server, mta-db.mail.nethere.net. To remove the ip address using a mta server:

<pre>
1. Log onto mta-1.mail
2. Switch to mysql
$mysql -h 10.0.0.69 -u policyd_inbound -p
username: policyd_inbound
password: p0stf1x!
3. Display database information
mysql>show databases;
4. Use policyd database
mysql>use policyd;
4. Display table information
mysql>show tables;
mysql>describe blacklist;
5. Verify ip address has been blacklisted
mysql>select * from blacklist where _blacklist='xxx.xxx.xxx.xxx';
6. Delete the entry
mysql>delete from blacklist where _blacklist='xxx.xxx.xxx.xxx';
7. Verify
mysql> select * from blacklist where _blacklist='xxx.xxx.xxx.xxx';
</pre>

The suspected mail server may also be in the helo table.

<pre>
mysql> describe helo ;
1. Check the IP address
mysql> select * from helo where _host='xxx.xxx.xxx.xxx' ;
2. Check the server helo name
mysql> select * from helo where _helo like '%server%' ;
3. Delete the entry
mysql> delete from helo where _host='xxx.xxx.xxx.xxx' ;
mysql> delete from helo where _helo like '%server%' ;
</pre>

====Add ip address to client_reject====

The client_reject list is one of the ways we combat spam. If we receive spam compaints from a mail server or otherwise identitfy a mail server as a source of spam, we can add it to a client_reject list. An email sent from a mail server on the client_reject list is rejected with a notice saying "Relaying denied due to excessive spam". To add an ip address to the list, we use admin-2. Once the list is updated on admin-2, it is pushed out to mta-1 and mta-2. To add an ip address on admin-2:

<pre>
Log into admin-2.nethere.net.
bash-3.00# cd /dist/files/nhmta/nethere/conf/postfix/common
bash-3.00# co -l client_reject
RCS/client_reject,v --> client_reject
bash-3.00# vi client_reject

-Add the ip address to the bottom of the list in the appropriate format. It you want to reject any email from the mail server, you would use:
xxx.xxx.xxx REJECT Relaying denied due to excessive spam
where xxx.xxx.xxx is the /24 subnet of the mail server the spam originated from. Do not place a period after the last octet. For example, to add a /16 subnet you would use:
xxx.xxx REJECT Relaying denied due to excessive spam
The REJECT key word tells postfix to reject the email. If you want to allow email from a /24 subnet, use the following format:
xxx.xxx.xxx OK
This tells postfix to accept email from that subnet.

bash-3.00# ci -u client_reject
RCS/client_reject,v <-- client_reject
new revision: 1.286; previous revision: 1.285
enter log message, terminated with single '.' or end of file:
>> .
done
bash-3.00# make
/nethere/software/nhmta/postfix-2.4.5/sbin/postmap -C . client_reject
mv client_reject.db maps/hash/client_reject.db
bash-3.00# cd /dist/rdist
bash-3.00# gmake nhmta-update
updating host mta-1.mail.nethere.net
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject
updating host mta-2.mail.nethere.net
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject
bash-3.00#
</pre>

====Free mysql disk space on mta-db.mail.nethere.net====

In response to nagios alert "[Nagios] PROBLEM alert - mta-db.mail.nethere.net/Disk mysql is WARNING", to clear space within the mysql database:

1. ssh to mta-db.mail.nethere.net.

2. Log into mysql and execute the following:

[root@mta-db user]# mysql -uroot -p

Enter password: dB@dm1N!

mysql> show databases;

mysql> use policyd;

mysql> show tables;

mysql> describe helo;

mysql> describe throttle;

mysql> describe throttle_from_instance;

mysql> select * from throttle_from_instance limit 10;

mysql> select now();

mysql> select unix_timestamp(now());

mysql> select count(*) from throttle_from_instance where _expire > 1426534697;

mysql> select max(_expire) from throttle_from_instance;

mysql> truncate table throttle_from_instance;

mysql> select count(*) from throttle_from_instance;

mysql> select * from throttle_from_instance;

mysql> show tables;

mysql> optimize table throttle_from_instance;

mysql> show databases;

mysql> use policyd;

mysql> quit

[root@mta-db user]# df -h

== Errors ==

=== Common webmail errors ===

Error:
--cut--
Fatal error: Call to undefined function: applicatio€”¬p() in
/www/webmail.nethere.net/htdocs/x/m/templates/message/navbar.inc on line 7
--cut--

Solution: Restart apache on webmail-1 (apachectl restart)

Error:
--cut--
ERROR There was an error sending your message: unable to add recipient
[webhosting@nethere.com]: Invalid response code received from server
--cut--

Solution: More than likely DNS/domain issue, but check mta-1.nethere.net to be sure:
--cut--
Aug 30 15:08:54 mta-1 nh/smtpd[50501]: NOQUEUE: reject: RCPT from webmail-1.mail.nethere.net[66.63.128.181]: 450 <orders@mulligrins.com>: Sender address rejected: Domain not found; from=<orders@mulligrins.com> to=<webhosting@nethere.com> proto=ESMTP helo=<webmail.nethere.net>
--cut--
Problem: Webmail shows "1-5" messages but inbox appears empty. POP shows messages #1,2,3,4,5... could not be retrieved... server response error cannot open the message file it's gone.

Solution: Ownership issue

1) Use the provisioning tool to determine mailbox location:
a) Click on email account, Engineering
b) Location is the "Storage Path:"

2) Log in to either mail server then change ownership of maildir to mailuser:mailuser
# chown -R mailuser:mailuser /nfs/#/platform/u/s/username

== DEPRECIATED INFORMATION ==

=== Webservers ===

==== Allowing SSI (Server Side Includes) ====

***** NOTE: This has been DEPRECIATED, all servers now allow SSI by default *****

We do not allow EXEC permissions for SSI.

1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>
# co -l www.domain.tld(.common)

2) Add the following lines under the <Directory> directive
--cut--
AddType text/html .shtml
AddHandler server-parsed .shtml
--cut--

3) Edit the "Options" line to be the following:
--cut--
Options Indexes FollowSymLinks IncludesNOEXEC
--cut--

4) Add a DirectoryIndex line above the <Directory> directive with the following:
--cut--
DirectoryIndex index.shtml index.html index.htm home.html home.shtml index.php
--cut--

** In summary, the config should look something similar to this **
--cut--
DocumentRoot "/www/www.domain.tld/htdocs"
DirectoryIndex index.shtml index.html index.htm home.html home.shtml
<Directory "/www/www.domain.tld/htdocs">
AddType text/html .shtml
AddHandler server-parsed .shtml
Options Indexes FollowSymLinks IncludesNOEXEC
Order allow,deny
Allow from all
</Directory>
--cut--

5) Check in Apache config
# ci -u www.domain.tld(.common) < /dev/null

6) Restart Apache
# apachectl configtest
# apachectl restart

More SSI notes can be found here:
http://httpd.apache.org/docs/1.3/howto/ssi.html

= SimplyInternet =

= Z-net =

= Migrating to LAMP Host Servers =

Below are outlines of the steps to migrate website & email hosting from nethere to lamphost configured servers. This is almost certainly incomplete at this stage. The exact steps to follow can vary depending on specifics of the client, so use this as a guide only. Be ready to adjust commands or add steps as needed.

== Edit Nethere DNS ==

ssh user@admin-1.nethere.net

sudo -i

ssh ns1.nethere.net

cd /named/master

co -l example.com

vi example.com

ci -u example.com

rndc reload example.com

Make sure to increment serial number by 1 each time a change is made.

== Migrate Website to nh1.jcihosting.net ==

Set TTL to 300 on ns1.nethere.net

Open up these web admins to get current account/site info:

http://toolbox.nethere.net/ -> Provisioning Tool (search for domain name)

https://secure.johncompanies.com/mgmt/index.html (search for col#####)

https://www.lamphost.com/admin/user/user/create

username: col#####

email: same as listed for col##### account

password: random, secure, won't be shared with client

defaults on rest of page

https://nh1.jcihosting.net:8443/

LAMP Host Users -> Add User -> col#####

Virtual Hosts -> Add Virtual Host

Username: Select same username as above from pulldown

Domain: example.com

Hosting Plan: select plan that matches what's listed in https://secure.johncompanies.com/mgmt/view.html?cid=col#####

Add default DNS: No

Virtual Hosts -> Site Manager (next to domain added)

Site Manager Users -> Add User

Username: can be anything - I typically use Optigold Login from nethere admin

Password: make up something secure and record - this one will go to the user

Email & FTP -> Add Account

Email Account: ftp

Password: make up something secure and record - this one will go to the user

Enable FTP Access: Yes

Optional: Add MySQL database if site needs it (WordPress etc)

ssh to nh1.jcihosting.net and run 'add_mysql.php example.com'

MySQL database info will be emailed to webmaster@lamphost.com (and output on command line - ignore instructions to run additional commands)

ssh nh1

cd /var/www/example.com/

rsync -v --archive --one-file-system --delete --delete-during --rsh=/usr/bin/ssh matt@unixweb-2.nethere.net:/www/www.example.com/htdocs ./

Change 'matt' to your username. Note that I had to add my username to groups httpd & webuser to allow me to copy website files without being root.

chown -R col#####:col##### htdocs

If site has MySQL data find the connection info. For example, WP sites will have the info in htdocs/wp-config.php

mysqldump -h mysqldb-1.webhost.nethere.net -u username --password=password databasename > databse.sql

mysql exampledotcom < database.sql

rm database.sql

If site is ready to be made live on nh1.lamphost.net then update DNS on ns1.nethere.com

Send email to client with details. Search support@jcihosting.com Sent folder for emails with Subject "updated hosting for" for example emails. You will have to adjust as needed (username/passwords, different info based on specific client).

== Migrate Email to nh1.jcihosting.net ==

If migrating website & emails both follow the above steps to transfer the website first.

If migrating emails only the follow the first section of the above steps to set up the virtualhost on nh1.

Set TTL for MX record to 300.

Generate email report:

Log in to http://toolbox.nethere.net/

Select Provisioning Tool and search for the domain

Navigate to the Email section and then Generate Report.

Select these fields for the report:

Mail address

Alias

Forwarding address

Mail storage path

Copy the result of the report and save it as a text file in /tmp/ on nh1.jcihosting.net named example.com.txt (replace domain name)

Make sure the virtualhost has enough email accounts allocated before proceeding. Edit the virtualhosts.virtualhost_settings table directly if need be to increase allocation.

Run this script once and only once:

/usr/local/bin/migrate_mail_accounts_from_nethere.php example.com

That will create all email accounts and aliases.

A file containing the list of email accounts and their new passwords will be created as /tmp/example.com-passwords.txt. Grab a copy of that for sending to the client.

Another file will be created that contains the bash commands to sync emails from the old server. This will be named /tmp/example.com-sync.sh

Copy the file somewhere like the /root/ home dir and add "#!/bin/bash" as the first line and change perms to 750. Now you can run this script to sync all mails from nethere to nh1. Run it immediately to get an initial sync of emails.

When ready to switch email hosting over run the sync script one more time, then update DNS on nh1.nethere.net.

If you'd like to sync again after the move (if there was a delay or you think there may have been incoming emails during the transtion) remove the '--del' option from the sync script and then re-run.

Search support@jcihosting.com Sent folder for messages with Subject "new email hosting for" for examples of emails to use for communicating with the client.

Once DNS has been pointed to the new server, go into the NetHere Toolbox http://toolbox.nethere.net/ and go to Domain -> General and then set Status to 'On Hold'.

NetHere

2016-05-02T22:41:08Z

64.149.103.158: /* Email Restoral */

= NetHere =

URL: http://nethere.com

<br>
Access to servers is restricted to admin-1.nethere.net. Access to admin-1.nethere.net is restricted to our office and nat.johncompanies.com.

<br>
JCI Hosting acquired the web-hosting customers of NetHere. (Previously, NetHere had aquired customers from Simply Internet, and Z-net).
Customer billing/info are being imported from NetHere to JCI database
URL: https://secure.johncompanies.com/mgmt/index.html

<br>
Nethere has several tools for management. These are restricted to only allow
access from the office, or a few admin's homes.

cacti [http://cacti.nethere.net]
nagios[http://nagios.nethere.net]
toolbox [http://toolbox.nethere.net]
helpdesk [http://helpdesk.nethere.net]
support FAQ [http://www.nethere.net/website/support]
knowledge base [http://support.nethere.net/kb]
phpmyadmin [https://phpmyadmin.nethere.net]

== Virtual Hosts ==

<pre>
66.63.129.106
admin-1.nethere.net
admin-2.nethere.net
pike.nethere.net- cacti, nagios- off
shark.nethere.net- ldap
ntdb-1.nethere.net

66.63.129.105
home-1.nethere.net
ntweb-11.nethere.net
ntweb-6.nethere.net
web-3.schedulecafe.com

66.63.129.104
eel.nethere.net- ldap
mta-1.mail.nethere.net
nscache-1.nethere.net
nsrbl-1.nethere.net
phoenix.nethere.net (ns-1.nethere.net)
relay-1.mail.nethere.net

66.63.129.103
andromeda.nethere.net (ns-2.nethere.net)
nscache-2.nethere.net
nsrbl-2.nethere.net
mailbox-2.nethere.net

66.63.129.102 (vm2.eng.nethere.net)
koi.nethere.net- off- realserver
prov-1.nethere.net- cp.nethere.net, cp.znet.net, cp.simplyweb.net
scribe.nethere.net- prov.nethere.net, scribe.nethere.net
tetra.nethere.net- nethere.com, znet.com, simplyweb.com
web-2.inboxhq.net- helpdesk
webmail-1.nethere.net

66.63.129.101 (c.host.nethere.net)
ahi.nethere.net- ldap
mta-2.mail.nethere.net
relay-2.mail.nethere.net
unixweb-8.nethere.net

66.63.129.91
lists-1.nethere.net
sb-2.nethere.net
mx-1.nethere.net- spooling server
mailx-1.nethere.net- web server outbound mail server

207.167.93.106- vmware6.eng
ntweb-2.nethere.net
unixweb-12.nethere.net (no web sites on server)
svn1.eng.nethere.net- off

207.167.93.108- vmware8.eng
mailx-2.nethere.net
mx-2.nethere.net

207.167.93.110- vmware10.eng
unixweb-11.nethere.net

66.63.129.2 - New HP DL360
fpweb-2.nethere.net
unixweb-6.nethere.net
mta-db.mail.nethere.net

66.63.129.4- vmware9.eng
mysqldb-1.webhost.nethere.net
</pre>

== Provisioning ==

=== Webservers ===

OS: FreeBSD 4.11-RELEASE-p10 #23

==== General ====

<pre>
unixweb-1 through 8

All have the same configuration except for the following exceptions:

unixweb-4: ZN Front Page server (depreciated)
unixweb-5: ZN Front Page, Miva Merchant server
unixweb-6: SI shared counter server (for SI sites): /www/lucy.inetworld.net/htdocs/cgi-bin/Count.cgi

Server Software Installed

Apache
# httpd -v
Server version: Apache/2.0.55
Server built: Apr 5 2006 17:04:01
PHP
# /usr/local/bin/php4 -v
PHP 4.4.2 (cgi-fcgi) (built: Mar 2 2006 09:31:57)

Miva Merchant (unixweb-5)
MySQL
# mysqladmin version -u root -p
Server version 4.0.25

Apache configurations are in the following directories:

/nethere/conf/apache/conf/nvhosts (name based hosting); naming convention is www.domain.tld
/nethere/conf/apache/conf/vhosts (ip based,SSL); naming convention is www.domain.tld (IP information) and www.domain.tld.common (general site information)

NetHere specific scripts are located in:
/nethere/sbin

Checking Server/site status via mod_status apache module:

# apachectl stop
# /usr/local/etc/rc.d/httpd.sh start-status

URL: http://unixweb-#/status (replace # with the unixeweb number)
** NOTE: After viewing status, do the following, otherwise other users can also view status by using .htaccess.
# apachectl stop
# apachectl startssl
</pre>

==== Provisioning ====
Provisioning new sites:

*** All website provisioning is done via NH script (webadd) on sawfish (admin-1.nethere.net) ***
# /nethere/sbin/webadd -h for usage

Note: We've discontinued new website provisioning on all servers except for unixweb-6, unless otherwise noted for domain

==== FTP Password Adjustment ====

===== Unix Server =====
To change an FTP password:

On the customers server as root do

passwd <username>

(The password prompt will not echo entry)

===== Windows Server =====

To change an FTP password:

<pre>
Find the users login (ie aa5014) in the provisioning tool.
Use Remote Desktop to get into the server.
On Desktop find FTP application Icon.
Use that tool to update the user's password.
</pre>

==== Domain Aliasing ====

For domain aliasing:

1) Edit the /named/named.master on phoenix (ns1.nethere.net)
a) Checkout the file; open with vi
# co -l named.master
# vi named.master
b) find the domain that's serving as the master domain, then, following the general format of the file to add the domain aliases under the master domain entry, using the master domain zone file
i.e.
--cut--
zone "domain.tld" { type master; file "master/domain.tld"; };
zone "aliased_domain.tld" { type master; file "master/master_domain.tld"; };
--cut--
c) save the file, then check-in:
# ci -u named.master < /dev/null
d) update the zone records, using the makefile in /named; check for errors
# make new-zone
# tail /var/log/named
**Note: some domains are IP based virtually hosted (i.e. SSL certificate). This requires a modification to the standard aliasing procedure. If this is the case, do the following
e) Create a new zone file called "domain.tld-alias" in /named/master, copy current domain.tld zone file to the domain.tld-alias file, and adjust the "website" records, removing whatever IP based information is there, and replacing with the appropriate $INCLUDE name-based host template.
e.g.
tropicalshade.net:
--cut--
;; ntweb-4
@ IN A 66.63.136.4
www IN A 66.63.136.4
--cut--
tropicalshade.net-alias
--cut--
;; website
$INCLUDE master/nvhost.ntweb-4
--cut--
f) Adjust the named.master zone file in /named accordingly:
i.e.
--cut--
zone "aliased_domain.tld" { type master; file "master/master_domain.tld-alias"; };
--cut--

2) Edit the Apache configuration for the domain on the web server the master domain is hosted on
a) Script is /nethere/sbin/webalias
# /nethere/sbin/webalias -h (for usage)
e.g. # /nethere/sbin/webalias -d www.originaldomain.tld -a www.domainalias.tld

==== CGI Scripting ====

For security, we've implemented suexec on all sites.
All CGI *must* be placed in the cgi-bin/ (and/or cgibin/ directory for unixweb-4, 5) directory

CGI permissions break down as follows:

Ownership: user=username, group=webuser
Permission: cgi-bin/ = 755; files = 755

suexec errors can be found here:
/www/default/logs/suexec_log

cgi errors for a domain can be found in the main apache error logs for the domain:
/www/www.domain.tld/logs/error_log

More information on CGI scripting can be found here:
http://httpd.apache.org/docs/1.3/howto/cgi.html

==== Formmail Provisioning ====

Formmail is provisioned via NH script (newformmail) on server domain is hosted on.
# /nethere/sbin/newformmail -h for usage

Usage: newformmail [-h] [-d domain] [-i ip_address] [-e "email1 email2"]

==== Disabling a site ====

To disable/enable a website, there is a script (webstatus) located on each webserver:
/nethere/sbin/webstatus -h for usage

1) Disable:
# /nethere/sbin/webstatus -r -d -s www.domain.tld

2) Enable:
# /nethere/sbin/webstatus -r -e -s www.domain.tld

==== Stats Provisioning ====

We utilize webalizer for statistics.

Stats are provisioned via NH scripts on server domain is hosted on.

1) Provision stats for the domain (webaddstats_unix):
# /nethere/sbin/webaddstats_unix -d www.domain.tld

2) Run stats for the domain (runstats_unix):
# /nethere/sbin/runstats_unix www.domain.tld

More information on Webalizer can be found here:
http://www.mrunix.net/webalizer
README: ftp://ftp.mrunix.net/pub/webalizer/README

==== Removing a site ====

1) Site removal is primarily done via a NH script (webdel) on the webserver the domain is hosted on:
# /nethere/sbin/webdel -h for usage
# /nethere/sbin/webdel -r -d www.tldomain.com

2) Update DNS accordingly (ns1.nethere.net)
a) Checkout /named/named.master
# co -l named.master
b) Remove line containing domain
c) Checkin /named/named.master
# ci -u named.master < /dev/null
d) Update zones
# make new-zone
e) Move zone files from /named/master to /named/archive
# mv domain

==== starter removal ====

All starter sites on home-1.nethere.net

1) Removal primarily done via a NH script (webdel_home) on home-1
# nethere/sbin/webdel_home -h for usage
i.e. webdel_home -p zn -s startername <== for removing znet

==== Disk Quota increases ====

Quotas are increased by using the "edquota" command:
# edquota username
Note that quotas are in KB, so a conversion is necessary. Also note that the "soft" quota is 1 MB less than the "hard" quota.

The calculations are as follows:
"soft" = ( quota(in MB) - 1 ) x 1024
"hard" = quota(in MB) x 1024
e.g. for a 500 MB quota:
soft => (500 - 1) x 1024 = 510976
hard => 500 x 1024 = 512000

To check a quota:
# quota -v username

You can also use the prototype users:

# edquota -p quota100 username (100 MB)
# edquota -p quota500 username (500 MB)
# edquota -p quota1000 username (1000 MB)

==== Granting of shell - Enterprise packages ====

Shell is granted by the "chsh" command. By default, C shell (csh) is used:
# chsh -s shell_needed username
e.g. # chsh -s csh username

*Note: customers must provide us with the static IP address(es) they will be connecting from. The IP address(es) must be added to the configuration on admin-1.
1) Modify the corresponding rules files on admin-1:
/dist/files/nhweb/etc/ipf.rules.fxp0 , .em0 (RCS controlled)
2) Push updates to servers
# cd /dist/rdist
# gmake nhweb
# gmake nhweb-update
3) Reload the rules on the corresponding server that needs access to
# ipf -Fa -f /etc/ipf.rules

==== SSL certificate installs ====

1) Whois the site for information on Registrant, email - to be used for SSL generation
# whois domain.tld

2) Create SSL self signed certificate, get CSR for customer to sign
a) SSL generation is done by script: /nethere/conf/apache/pki/newsslcert.sh
b) Copy CSR for customer

3) Create/update Apache configuration files via NH script (webadd_ssl)
# /nethere/sbin/webadd_ssl -h for usage
a) webadd_ssl [-h] [-d domain] [-s ssl_domain] [-n]
# webadd_ssl -d www.domain.tld -s www.domain.tld

4) Update DNS zone with new IP address - done on ns1.nethere.net
a) Check out DNS zone file in /named/master
# co -l domain.tld
b) Adjust A records:
--cut--
;; unixweb-## << enter the unixweb server number for ease of ID
@ IN A vhost_ip_address
www IN A vhost_ip_address
--cut--
c) Check in DNS zone file
# ci -u domain.tld < /dev/null
d) Reload zone file
# rndc reload domain.tld
e) check /var/log/named for errors

5) Update DNS PTR record for IP address - done on ns1.nethere.net
Note: Assuming IP address a.b.c.d
a) Check out in-addr.arpa zone for IP address in /named/master
# co -l a.b.c
b) Follow format for PTR records --cut--
d IN PTR www.domain.tld.
--cut--
c) Check in PTR zone file # ci -u a.b.c < /dev/null
d) Reload PTR zone
# rndc reload c.b.a.in-addr.arpa
e) check /var/log/named for errors

==== Updating SSL certificate ====

1) cd to /nethere/conf/apache/pki/ssl.crt on server site is hosted on

2) Check out (RCS) www.domain.tld.crt file
# co -l www.domain.tld.crt

3) Edit the file, remove old certificate, paste in new certificate

4) Check in (RCS) the www.domain.tld.crt file
# ci -u www.domain.tld.crt < /dev/null

5) Check, restart Apache
# apachectl configtest
# apachectl stop
# apachectl startssl

6) Verify httpd started:
# ps auxw | grep httpd
a) If no processes, will need to revert back to old SSL cert, and restart apache. Check logs for errors
# view /www/default/logs/ssl_engine.log
b) Check for "Unable to configure RSA server private key" and "key values mismatch" entries - this means a bad SSL certificate

==== Password protection ====

Http (simple) password protection is governed by the Apache configuration for the domain

1) Create userdb, users file in the domain root directory (/www/www.domain.tld):
# mkdir userdb
# cd userdb
# htpasswd -bc users username password

2) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>
# co -l www.domain.tld(.common)

3) Edit Apache configuration, add the following lines within the VirtualHost container
--cut--
<Directory "/www/www.domain.tld/dir_to_be_protected">
AuthType Basic
AuthName "www.domain.tld/dir_to_be_protected authentication"
AuthUserFile /www/www.domain.tld/userdb/users
<Limit GET POST>
require valid-user
</Limit>
</Directory>
--cut--

4) Check in Apache config
# ci -u www.domain.tld(.common) < /dev/null

5) Restart Apache
# apachectl configtest
# apachectl restart

More on http (simple) password protection can be found here:
http://httpd.apache.org/docs/1.3/howto/auth.html#basic

==== .htaccess ====

Used if customers want control of certain Apache directives (i.e. Authentication, etc.)

1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>
# co -l www.domain.tld(.common)

2) Add the AllowOverride directive in the <Directory> section, under the PHP FCGIWrapper
i.e.
<Directory "/www/www.domain.tld/htdocs">
FCGIWrapper /www/www.domain.tld/htdocs/cgi-bin/php4 .php
AllowOverride AuthConfig FileInfo Indexes Limit
</Directory>

3) Check in Apache config
# ci -u www.domain.tld(.common) < /dev/null

4) Restart Apache
# apachectl configtest
# apachectl restart

More info on Allow Override can be found here: http://httpd.apache.org/docs/1.3/mod/core.html#allowoverride

==== Domain re-provisioning ====

## NOTE: Following is for domain being provisioned on same server (i.e. just being renamed)

1) Adjust DNS - ns1.nethere.net
a) Rename the DNS zone file to the new domain, remove the old DNS zone files from /named/master and /named/master/RCS
b) Check in the new DNS zone file
c) Edit /named/named.master - replace the old domain with the new one, alias as necessary
d) Reload DNS zones - make new-zone in /named

2) Adjust Apache config - server domain is hosted on
a) Find current config file(s) in /nethere/conf/apache/conf/<vhosts,nvhosts>
b) Replace old domain name entries with new domain name, alias as necessary
c) Save as new_domain.tld
d) Remove old_domain.tld(.common), RCS/old_domain.tld(.common)
e) Check in new_domain.tld via RCS:
# ci -u new_domain.tld < /dev/null
f) Check out Apache include configuration file via RCS /nethere/conf/apache/conf/<nvhosts.conf,vhosts.conf>
g) Edit nvhosts.conf or vhosts.conf, replace old_domain.tld entries with new_domain.tld
h) Check in via RCS /nethere/conf/apche/conf/<nvhosts.conf,vhosts.conf>

3) Rename directory for new domain
# cd /www
# mv www.old_domain.tld www.new_domain.tld

4) Adjust PHP stub files/configuration
a) Adjust php.ini file in /www/www.domain.tld/(php4,php5)
b) Adjust PHP stub files in /www/www.domain.tld/htdocs/cgi-bin/(php4,php5)
*note: need to chflags to "noschg" for /www/www.domain.tld/htdocs/cgi-bin/(php4,php5) before being able to update the stub files
after adjusting, be sure to chflags schg /www/www.domain.tld/htdocs/cgi-bin/(php4,php5)

5) Edit password file (vipw), replace old_domain.tld entries with new_domain.tld
# vipw

6) Restart Apache
# apachectl configtest
# apachectl restart

## NOTE: Following is for domain being re-provisioned on new server

1) Run NH script "webadd" on sawfish to provision domain on new server, *DO NOT* reload DNS, use same user/pass as before.
2) After customer has uploaded site to new server and gives the ok do the following:
a) Update DNS zone for domain.tld; reload zone for domain.tld
b) Wait 48 hours, then remove the site off the old server
# /nethere/sbin/webdel

==== FTP space provisioning ====

Generally FTP sites are provisioned on the server that hosts the main website. In the case of NT based FTP sites, we usually provision them on the server with the most space available.

1) Provision site on server
a) Done via NH script (webadd_ftp): webadd_ftp [-h] [-d domain] [-u username] [-p password] [-n]
# /nethere/sbin/webadd_ftp -h <- for usage
EX: for ftp.domain.tld:
# /nethere/sbin/webadd_ftp -d ftp.domain.tld -u username -p password
(leaving off the -n will restart the proftpd process)
b) Note: The host IP address will be given when the provisioning is completed, use that ip (a.b.c.d) for DNS entries
c) Note: If this is an existing customer on the server, you'll need to increase the quota manually by 100 MB for the customer, see the section on quota increases for more info.

2) Adjust DNS for domain on ns1.nethere.net
a) Checkout zone for domain
b) Add ftp host entry for domain, adjust serial
EX for domain.tld:
--cut--
ftp IN A a.b.c.d
--cut--
c) Check in zone for domain
d) Reload zone
# rndc reload domain.tld
3) Adjust PTR record for domain, adjust serial
a) Checkout zone for a.b.c
b) Add record for domnain:
--cut--
d IN PTR ftp.domain.tld.
--cut--
c) Check in zone for a.b.c
d) Reload zone
# rndc reload c.b.a.in-addr.arpa
4) Check for DNS errors
a) tail /var/log/namedb

==== SiteBuilder provisioning ====

unixweb-7.nethere.net

*Must re-provision site on unixweb-7.nethere.net (if not already done)

1) Log in to SB admin:
http://sitebuilder.nethere.net/admin
a) username: root

2) Add site to SB config
a) Click on Site Management -> Add regular
* Alias is website username: i.e. aa####
* Check the "Active" box
* Plan is "BasePlan"
* Password same as website
b) Click on "Publish Properties
* Check "Allow publishing"
* Site host name: www.domain.tld
* FTP host: unixweb-7.nethere.net
* FTP login/password: same as site user/pass
* FTP working directory: leave blank
c) Click Apply

==== Name Servers ====

OS: FreeBSD

==== General ====

ns1.nethere.net - Primary name servers for DNS zone records
/named - contains the files that have all DNS domain zone entries (named.master, named.slave, named.acl) plus Makefile for distributing DNS records
/named/master - contains all the domain zone files for which we are authorative for, as well as IP address (PTR records)

ns2.nethere.net - Secondary (slave) name server for DNS zone records

nsrbl-1.nethere.net - RBL (Realtime Blackhole List) DNS server
/named/rbldns/cache - contains the files for domains that we specifically allow or deny

nscache-1,2 - caching name servers

==== Adding DNS website entries ====

1) Create a DNS zone file for domain.tld via NH script (zoneadd_vhost)
# /nethere/sbin/zoneadd_vhost -h (for usage)

==== Adding DNS IP entries ====

1) Create the forward and reverse DNS records via NH script (zone_generate) for a netblock, will create /tmp/customer.forward and /tmp/customer.reverse files to be read
a) /nethere/sbin/zone_generate -h for usage
i.e. for netblock a.b.c, starting IP d, ending IP z
# zone_generate -n a.b.c -b d -e z -p customer

2) Checkout, edit the IP in-addr.arpa zone file, reload the zone
a) # co -l a.b.c
b) Seach for the nearest netblock area for the domain, follow format for customer info, read in the /tmp/abbrev.rdns file accordingly, increase Serial for zone in YYYYMMDD## format
e.g.
--cut--
;;;;
;; 66.63.152.232/30 (255.255.255.252)
;; Description: First Choice Home Improvement
;; Contact: Shannon Hill <firstchoicehi@hotmail.com>, (858) 277-5351
;; Location: AR-1, Serial3/0/18:0
;;;;
232 IN PTR firstchoice-net.access.nethere.net.
233 IN PTR firstchoice-gw.access.nethere.net.
234 IN PTR firstchoice-2.access.nethere.net.
235 IN PTR firstchoice-bcast.access.nethere.net.
--cut--
c) # ci -u a.b.c < /dev/null
d) # rndc reload c.b.a.in-addr.arpa
e) verify loading of zone: # tail /var/log/named
3) Checkout, edit the forward DNS zone file, reload the zone
a) # co -l access.nethere.net
b) Search for the nearest neblock area for the domain, follow format for customer, read in the /tmp/abbrev.fdns file accordingly, increase Serial for zone in YYYYMMDD## format
e.g.
--cut--
;; 66.63.152.232/30 (255.255.255.252)
firstchoice-net IN A 66.63.152.232
firstchoice-gw IN A 66.63.152.233
firstchoice-2 IN A 66.63.152.234
firstchoice-bcast IN A 66.63.152.235
--cut--
c) # ci -u access.nethere.net < /dev/null
d) # rndc reload access.nethere.net
e) verify loading of zone: # tail /var/log/named

==== Unblocking RBLd IP addresses ====

There are two primary reasons why we add customers to the allow relay list:
a) They have a static IP address (i.e. DSL, T1) and wish to use our mail servers
b) They've been blocked by one of our subscribed blackhole lists, however, have patched their machine and are no longer open to relay.

To do this on nsrbl-1.nethere.net:

1) Checkout the allow.relays.nethere.net file located in /named/rbldns/cache

2) Edit the file, and add the IP address in the following format:
--cut--
a.b.c.d YYYYMMDD hostname reason for listing
--cut--

You can also add subnets via '/' notation for relay
--cut--
a.b.c.d/28 YYYYMMDD hostname reason for listing
--cut--

3) Check in the file

4) Updates to the rbldns zone are done automatically on the hour, every hour, so no need to do anything else.

Note:

For "permanent" (i.e. customer static IP addresses), add the IP address in the "## permanent allowed relay (i.e. customer w/ static IP)" section
For "temporary" (i.e. blackhole listed IP addresses), add the IP address in the "## temporary" section

Also, for temporary IP addresses, we need to send the note to the requester detailing the following:
a) If the IP address relays Spam/UCE/Viruses through us, it is to be removed permanently.
b) The customer must follow the steps listed on the blackhole list the IP was listed on to get removed.

==== Flushing DNS cache for a domain ====

Due usually to a bad zone or excessively long TTL for a domain, the cache for it will need to be flushed.

On nscache-1:
# /usr/local/sbin/rndc flushname domain.tld

If cache is still corrupted (i.e. zone lookups either fail or are incorrect), need to stop and restart the caching server:

# /etc/init.d/local.named stop
# /etc/init.d/local.named start

==== SiteBuilder ====
http://sitebuilder.nethere.net/
-URL used to test sitebuilder

http://sitebuilder.nethere.net/Login
-URL for control panel.
<p>This is where the customer also logs in to manage
their web site and also where you log in to administer sitebuilder.
Sitebuilder is hosted on sb-2.nethere.net. The site is designed/built on
this server and published to unixweb-7.nethere.net. This is the only server
that can host a sitebuilder web site. The admin log in to manage
sitebuilder is:

Username: admin
Password: N3tH3r31!

==== Cart32 ====
cart32 information:

Cart32 is hosted on ntweb-6.nethere.net. That is where all of the
configuration files are located. The location of the ini file is
D:\websites\Cart32cgi/cart32.ini. This is where ip restrictions to admin
panel are set, password can be reset for admin, time limit restriction
reset, etc.

There are four customers that still use cart32:

<pre>
https://www.cart.simplyweb.net/lab400/cart/c32web.exe
https://www.cart.simplyweb.net/nutragenics/cart/c32web.exe
https://www.cart.simplyweb.net/retrogen/cart/c32web.exe
https://www.cart.simplyweb.net/stonesculptorssupplies/cart/c32web.exe
</pre>

The client codes are lab400, nutragenics, retrogen, and
stonesculptorssupplies. You can reset their passwords through the admin
panel.

To administer cart32:

Control Panel: https://www.cart.simplyweb.net/cart/c32web.exe/Admin
Username: administrator
Password: N3tH3r31!
Cart Admin Password: N3tH3r31!

=== Webmail ===

==== Removing webmail filters ====

All webmail filters are stored in the MySQL database on webmail-1.

Simple SQL commands:

* Choosing a DB
mysql> use db_name;

* Showing tables in a DB (must be using a DB)
mysql> show tables;

* Describing a table's fields
mysql> describe table_name;

For NetHere, the database is horde_nh.
For Simply, the database is horde_si.
For zNET, the database is horde_zn.

1) Log to MySQL as root, using the normal root password
a) # mysql -u root -p
2) Select the appropriate database
a) mysql> use horde_nh
3) Delete webmail filters.
a) mysql> delete from horde_prefs where pref_uid='<username>' and pref_name='filters';
Replace <username> with the users login.

==== Removing Address book entries ====

1) Log in
2) Select appropriate db
3) Find object_id, delete
a) mysql> select object_id from turba_objects where object_email='email@domain.tld';
b) mysql> delete from turba_objects where object_id='object_id_obtained_previously';

==== Repairing webmail address book ====

1) Export (via webmail, if possible) the address book to a csv file.

2) Delete the Turba objects in the mySQL database in the turba_objects table that is assigned to the customer.
mysql> delete from turba_objects where owner_id='username';

3) Import the saved address book via webmail.

==== Changing webmail From Information ====

1) Login as the customer (http://webmail.nethere.net)

2) Click on Options

3) Click on Personal Information

4) Select either default identity or a new one and click on Edit Your Identities

=== MySQL ===

All unixweb-## boxes have MySQL processes running on them. DB provisioning is done on the same server as which hosts the domain.
For Windows 2000 MySQL provisioning, all DBs are placed on ntdb-2.nethere.net

==== MySQL DB provisioning ====

Use https://phpmyadmin.nethere.net
User: root
PW: <root db pw>

1) Select the server that the DB needs to be hosted on, typically the same server that the website is hosted on.
2) Create the DB
a) Under "MySQL" - "Create new database" - the DB name is typically the website username; click "Create"
3) Create the User, set permissions for user on DB
a) Click "Home" (upper left), then click "Privileges" (Under "MySQL")
b) Click "Add a new User" make sure "Any host" selected, "User name" is typically the website username, "Password" is typically the website password. !LEAVE "Global Privileges" settings deselected! Click "Go" (bottom right corner) when finished.
c) Under "Database-specific privileges" be sure to add the user database
d) Select DB, then for "Privileges:" make sure the following are checked:
Select, Insert, Update, Delete, Create, Alter, Index, Drop, Create Temporary Tables, Lock Tables, References

=== List Server ===

lists-1.nethere.net

==== Mail List provisioning ====

1. Go to the /mailman/bin directory:
# cd /mailman/bin

2. Create the list on lists-1:
# ./newlist -q \
listname@lists.domain.com \
mailman-owner@lists.domain.com \
password

3. Configure the list using default settings:
# ./config_list -i /mailman/data/defaultlist.cfg listname

*** IF A NEW DOMAIN FOLLOW INSTRUCTIONS BELOW ***

NOTE: All files in /etc/mail are controlled by RCS

4. Go to the /etc/mail directory:
# cd /etc/mail

5. Add the list domain to the mailertable file:
lists.domain.com mailman:lists.domain.com

6. Add the list domain to the relay-domains file.

7. Add the list domain to the virtuserdomain file.

8. Add the following entries to the virtusertable file:
mailman@lists.domain.com <customer_email_address>
mailman-owner@lists.domain.com mailman@lists.domain.com

9. Run make to rebuild all files:
# make

10. Run make to restart sendmail:
# make restart

=== Backup Server ===

backup-1 -> odd numberd unix
backup-2 -> even numbered unix

On Saturdays, we do a level 0 dump
the rest are differential

Restoration fees are basically $25 per day restored for email, $150 for website from cancelled archive, $25 a day for website

Restoration is based on customer request. In order to restore a site/mailbox fully, you must start restoration from the previous level 0 backup and continue until the day after the day requested, since backups are performed in the morning.

General rule is to leave the gzipped file available for 24 hours, after that remove it.

==== Site Restoral ====

If the site was removed by the web_del script, then so long as it is within 30 days since removal, the gzipped site will be available within /www/archive/www.domain.tld-date_removed.tar.gz
Extract with tar:
# tar -zxvpf www.domain.tld-date_removed.tar.gz

If the site needs to be restored from backup, here are the steps

1) To determine where backups are stored, go to /nethere/conf/backup

2) grep for the server within the directory to determine the backup directory used
i.e.
# grep unixweb-1 *

4) Go to /backup1/unix/unixweb-1.nethere.net
a) you'll see a bunch of directories, named <date>-<dump_level>

5) Since the backups happen in the morning, typically you'll need to go to the day before, restore
e.g. for 11/10
a) cd 20041109-3
b) restore -if and whatever the filesystem is named
(for interactive: add files/dirs, extract, then use 1 for volume #)
i.e.
restore> add <path_to_dir/files>
restore> extract
(for volume #: 1)
(Set owner: y)

6) After you've restored the files, just use scp as root on sawfish to copy them off of backup-# and then onto unixweb-#
a) (on sawfish - two step process)
# scp backup-1:/tmp/<restored_file> /tmp
# scp /tmp/<restored_files> unixweb-1:/tmp

7) Cleanup any restored files off of backup-# server and sawfish

With the newly restored files, following are the steps necessary to restore a site:

1) Restore www.domain.tld directory to /www
# mv /www/archive/www.domain.tld /www

2) Restore configuration files
a) Apache: mv /www/archive/nethere/conf/apache/(n)vhosts/www.domain.tld /nethere/conf/apache/conf/(n)vhosts/
b) Webalizer: mv /www/archive/nethere/conf/webalizer/unix/www.domain.tld /nethere/conf/webalizer/unix

3) Add user to /etc/master.passwd:
a) vipw
(go to end of file, read in the master.passwd file from /www/www.domain.tld)
:r /www/www.domain.tld/master.passwd
b) Remove /www/www.domain.tld/master.passwd

4) Change flags on cgi-bin/ directories:
# chflags sunlnk /www/www.domain.tld/htdocs/cgi-bin/
# chflags schg /www/www.domain.tld/htdocs/cgi-bin/php*

5) Add configuration file entry to nvhosts/vhosts.conf
# co -l (n)vhosts.conf
# vi (n)vhosts.conf
# ci -u (n)vhosts.conf

6) Check apache, restart; verify apache
# apachectl configtest
# apachectl graceful
#

7) Cleanup remaining restored files
# rm -r /www/archive/www
# rm -r /www/archive/nethere

8) Enter zone into DNS on ns1.nethere.net
a) zone file
# mv /named/archive/domain.tld /named/master
# mv /named/archive/domain.tld,v /named/master/RCS
b) named.master file
# co -l /named/named.master
(add zone)
# ci -u /named/named.master
# cd /named; make new-zone
# tail /var/log/named (look for errors)
*************************************************
(12:40:47 PM) Henry Chan: restore is now available that is compatible with the 4.4bsd format
(12:41:04 PM) Henry Chan: to restore, use the following command: restore -c -i -f path_to_archive
(12:41:08 PM) Henry Chan: the "-c" is what does it
(12:41:29 PM) Henry Chan: (only applies to backup-1-new... doesn't work on backup-2 or backup-4)

==== Email Restoral ====
<pre>
Storage path prefixes:
/nfs/1 is sndg-netapp-1 (on backup-4 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)
/nfs/2 is sndg-netapp-2 (on backup-3 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)
/nfs/3 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)
/nfs/4 is sndg-netapp-1 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)
/nfs/5 is sndg-netapp-2 (on backup-1 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)
/nfs/6 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)

Looking up a mailbox's Storage Path:
You will need to know which directory and NFS server the customer's mail is
stored on. Use the Provisioning Tool to get the Storage Path (find the customer's mail
account, then click on the Engineering sub-tab under the Email tab). You should end up
with something like "/nfs/3/nh/h/t/htchan/Maildir". Match the prefix of the storage
directory with an NFS server (see prefixes above).

1) If restoring a recently deleted mail account:
a) Look for the archive on mailbox-1 in /nfs/archive/mail/{platform}/{username}.{date}-{PID}.tar.gz. If it doesn't exist, it's been too long and the only way to get email back is to restore from backup.
b) Make sure the account is re-created in Provisioning Tool and look up the storage path (see "Looking up a mailbox's Storage Path" above).
c) Extract the archive to a temporary directory:
Sample command:
cd /tmp; tar xzvpf /nfs/archive/mail/nh/zella.20090902-7508.tar.gz
Sample output:
nfs/2/nh/z/e/zella/
nfs/2/nh/z/e/zella/Maildir/
nfs/2/nh/z/e/zella/Maildir/tmp/
nfs/2/nh/z/e/zella/Maildir/new/
nfs/2/nh/z/e/zella/Maildir/cur/
nfs/2/nh/z/e/zella/Maildir/maildirsize
nfs/2/nh/z/e/zella/Maildir/.Trash/
nfs/2/nh/z/e/zella/Maildir/.Trash/tmp/
nfs/2/nh/z/e/zella/Maildir/.Trash/new/
nfs/2/nh/z/e/zella/Maildir/.Trash/cur/
nfs/2/nh/z/e/zella/Maildir/.Trash/maildirfolder
nfs/2/nh/z/e/zella/Maildir/.Drafts/
nfs/2/nh/z/e/zella/Maildir/.Drafts/tmp/
nfs/2/nh/z/e/zella/Maildir/.Drafts/new/
nfs/2/nh/z/e/zella/Maildir/.Drafts/cur/
nfs/2/nh/z/e/zella/Maildir/.Drafts/maildirfolder
nfs/2/nh/z/e/zella/Maildir/.Sent Items/
nfs/2/nh/z/e/zella/Maildir/.Sent Items/tmp/
nfs/2/nh/z/e/zella/Maildir/.Sent Items/new/
nfs/2/nh/z/e/zella/Maildir/.Sent Items/cur/
nfs/2/nh/z/e/zella/Maildir/.Sent Items/maildirfolder
nfs/2/nh/z/e/zella/Maildir/courierpop3dsizelist
Note:
In the above output, notice that the Maildir is located in "nfs/2/nh/z/e/zella", it will be used in the next step.
d) Copy the Maildir files to the new mailbox storage path using tar (tar does a better job at preserving things like symbolic links and permissions than cp or mv):
Sample command:
cd /tmp/nfs/2/nh/z/e/zella; tar cf - Maildir | (cd /nfs/3/nh/z/e/zella; tar xvpf -)
Output:
You should list the files as it is copied. Basically, similar to what you saw in step "c" above but without the "nfs/2/nh/z/e/zella" prefix.
Note:
The command format is basically:
cd /tmp/{restored-maildir-path-see-note-in-step-c}; tar cf - Maildir | (cd {new-storage-path-without-Maildir}; tar xvpf -)
e) Clean up temporary directory:
Sample command:
cd /tmp; rm -rf nfs

2) If restoring a mailbox to a particular date's backup:
a) Look up the storage path to the customer's mailbox and determine which backup server the dump file is on (see "Looking up a mailbox's Storage Path" and "Storage path prefixes" above).
b) Restore the level 0 and any incremental backups (in order) to /tmp on the backup server:
Sample commands:
cd /tmp
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090912-0/mail.dump
chflags -R 0 /tmp
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090913-1/mail.dump
chflags -R 0 /tmp
...skipped repetitive stuff here...
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090918-6/mail.dump
chflags -R 0 /tmp
Note:
After each restore, we need to recursively remove all flags from /tmp to eliminate the immutable flag that gets set on the files (this happens only on NetApp dumps... who knows why).
c) Create a new tar of the Maildir directory so that it can be copied to mailbox-1 for further processing:
Sample command:
cd /tmp/nh/z/e/zella; tar cf /tmp/archive.tar Maildir
d) Clean up /tmp:
Sample command:
rm -rf /tmp/nh
Note:
Depending on the platform, it might be /tmp/nh, /tmp/si, or /tmp/zn.
e) Copy the /tmp/archive.tar file to /tmp on mailbox-1.
f) On mailbox-1, extract the Maildir archive on top of what they already have:
Sample command:
cd /nfs/2/nh/z/e/zella; tar xvpf /tmp/archive.tar
g) Remove /tmp/archive.tar.
</pre>

==== Beginning of old instructions ====
Use the same techniques as site restoration, with the following exceptions:

1) Since everything is in maildir format, you have to reassmble things in /tmp/<username>, tar it up, and then restore it on any of the mailbox machines
a) using the following syntax:
# tar -zcpf /tmp/username.tgz yyyymmdd-#/platform yyyymmdd-#/platform yyyymmdd-#/platform
i.e.
# tar -zcpf /tmp/username.tgz 20050915-5/nh 20050914-4/nh 20050910-0/nh

2) In regards to mail spools, you'll need to use the chflags command to adjust the flags on the files. For whatever reason, they are stored with a system immutable flag
a) Do chflags -R noschg <dir> on the restored directory before copying the files with sawfish otherwise, the files cannot be deleted:
i.e.
# chflags -R noschg /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/
b) After tar/gzipping, remove the restored directories
i.e
# rm -r /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/nh

3) Can restore quickly with script on mailbox-1 (or mailbox-2) using the NH script "restoremail"
a) # /nethere/sbin/restoremail -h for usage
i.e.
# restoremail -p nh -f nhusername.tgz -n 2 -u nhusername
*** End of old instructions ***

* Backup locations *

backup-1:
backup-3.nethere.net
fpweb-1.nethere.net
home-1.nethere.net
koi.nethere.net
marmaduke.inetworld.net
ntdb-1.nethere.net
ntweb-1.nethere.net
ntweb-3.nethere.net
ntweb-5.nethere.net
ntweb-7.nethere.net
phoenix.nethere.net
unixweb-1.nethere.net
unixweb-3.nethere.net
unixweb-5.nethere.net
unixweb-7.nethere.net
shark.nethere.net
tetra.nethere.net
wms-1.nethere.net

backup-2:
andromeda.nethere.net
backup-4.nethere.net
eel.nethere.net
fpweb-2.nethere.net
ntdb-2.nethere.net
ntweb-2.nethere.net
ntweb-4.nethere.net
ntweb-6.nethere.net
unixweb-2.nethere.net
unixweb-4.nethere.net
unixweb-6.nethere.net
ds.znet.com
mx1.znet.com
mx2.znet.com
mx3.znet.com
dmx.znet.com
la.znet.com
uf.znet.com

backup-3:
backup-1.nethere.net
lists-1.nethere.net
mailbox-1.mail.nethere.net
mailbox-3.mail.nethere.net
mta-1.mail.nethere.net
mx-1.nethere.net
nscache-1.nethere.net
nsrbl-1.nethere.net
pegasus.nethere.net
scan-1.mail.nethere.net
sndg-netapp-2.nethere.net
relay-1.mail.nethere.net
webmail-1.mail.nethere.net

backup-4:
ahi.nethere.net
backup-2.nethere.net
cp-1.nethere.net
cygnus.nethere.net
dragon.nethere.net
lisa.nethere.net
mailbox-2.mail.nethere.net
mta-2.mail.nethere.net
news-1.nethere.net
nscache-2.nethere.net
nsrbl-2.nethere.net
sawfish.nethere.net
scan-2.mail.nethere.net
scribe.nethere.net
sndg-netapp-1.nethere.net
relay-2.mail.nethere.net
urchin.nethere.net

=== Mail servers ===
==== Organization ====
<pre>

NetHere Mail Servers
====================

Customers Inbound Mail Outbound Mail
/\ || || /\
|| || || smtp ||
|| || \/ ||
|| || ServerIron ||
|| || || ||
|| || || ||
|| || smtp \/ ||
|| || =========> mta-1 mta-2 ====> relay-1 <==== unixweb-* servers
|| || || /\ relay-2 ntweb servers
http || pop3 || || ||
|| imap || \/ +--------------+
|| || ServerIron |
|| || || |
|| || || |
|| || \/ |
|| || nsrbl-1 nsrbl-2 |
|| || (spamhaus, dcc blacklist) |
|| || || |
|| \/ \/ |
\/ mailbox-1 <=== scan-1 scan-2 |
webmail <===> mailbox-2 (sendmail/amavisd/sophos) |
imap mailbox-3 |
/\ /\ |
|| || \/
nfs || +---------------------------> mta-db
|| (LDAP)
\/
sndg-netapp-1
sndg-netapp-2
sndg-netapp-3

</pre>

==== Overview ====
mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.
mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.

nsrbl-1.nethere.net- Realtime blacklist check using dccd and spamhaus. The mta servers query the ip address of the mail server against the blacklist before transferring the email to the scanning servers.

nsrbl-2.nethere.net

scan-2.mail.nethere.net- Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.
scan-2.mail.nethere.net Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.

mailbox-1.mail.nethere.net- Front end server for the mail directories. Queries LDAP server for miscellaneous customer information including the directory the email should be stored in, passwords, spam sensitivity levels, etc. before delivering email to appropriate directory.
mailbox-2.mail.nethere.net
mailbox-3.mail.nethere.net

Mail is stored in maildir format:

/nfs/<nfs_number>/<platform>/u/s/username

sndg-netapp-1-new.nethere.net- Solaris system using zfs set up as a nfs mount on mailbox-1, -2, and -3. Email is stored on these platforms for all customers.
sndg-netapp-2-new.nethere.net
sndg-netapp-3-new.nethere.net

mx-1.nethere.net- spooling mail server

mailx-1.nethere.net- all outbound mails originating from a web server is sent out this email server. All web servers are set to use mailx.nethere.net which resolves (through the serveriron) to either mailx-1.nethere.net and mailx-2.nethere.net. Since we shut down mailx-2, all outbound mail from the web servers goes through mailx-1. This was to prevent all the customers email from being blacklisted as a result of a web site being compromised.

mta-db.mail.nethere.net- database server for the mta servers. All postfix databases are stored here.

webmail.nethere.net- Hosts the webmail program for webmail.nethere.net, webmail.simplyweb.net and webmail.znet.net.

==== Client mail settings ====

Inbound Mail Server:

POP3
pop3.nethere.net
pop3.znet.net
pop3.simplyweb.net

IMAP
imap.nethere.net
imap.znet.net
imap.simplyweb.net

Outbound Mail server:

mail.nethere.net
mail.znet.net
mail.simplyweb.net

Webmail:

webmail.nethere.net
webmail.znet.net
webmail.simplyweb.net

Control Panel:

cp.nethere.net
cp.znet.net
cp.simplyweb.net

==== nsrbl Disk Quota Warning ====

In the event you get a nagios regarding space on the two nsrbl servers, it is most likely the result of the dccd databases getting too large. To clear out space, log into each nsrbl server and execute the following:

df -h

cd /var/dcc/libexec

./stop-dccd

cd /var/dcc

rm dcc_db*

sync

sync

cd /var/dcc/libexec

./start-dccd

ps auxw | grep dccd

====Update outbound quota for a single ip address====

We restrict the total number of emails that a customer can send to 2000 email recepients per 24 hour period. This is tracked by the ip address of the computer that connects to the mta server when it sends an outbound emal. The ip address and count is stored in the postfix database server, mta-db.mail.nethere.net. To increase the quota for a single ip address, you can log into either mta-1, mta-2, or console direcly into the database server. To use one of the mta servers:

<pre>
1. Log onto mta-1.mail
2. Switch to mysql
$mysql -h 10.0.0.69 -u policyd_outbound -p
username: policyd_outbound
password: p0stf1x!
3. Review database information
mysql>show databases;
4. Switch to the policyd table.
mysql>use policyd;
5. Review table header information
mysql>show tables;
mysql>describe throttle;
6. Verify the ip address is above quota
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx'; #this is the ip address of the account you are increasing.
7. Increase the maxium number of recepients the ip address can send to
mysql>update throttle SET _rcpt_max = _rcpt_max + 20000 where _from='24.249.205.66';
8. Verify
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx';
</pre>

====Remove a mail server from policyd blacklist====

The ip address of a mail server can be placed on a blacklist for several reasons, to high of rate of inbound email from a single ip address, incorrectly formatted helo, etc. In the event you need to remove a mail server's ip from the policyd blacklist, you can log into either mta server of connect directly to the policyd database server, mta-db.mail.nethere.net. To remove the ip address using a mta server:

<pre>
1. Log onto mta-1.mail
2. Switch to mysql
$mysql -h 10.0.0.69 -u policyd_inbound -p
username: policyd_inbound
password: p0stf1x!
3. Display database information
mysql>show databases;
4. Use policyd database
mysql>use policyd;
4. Display table information
mysql>show tables;
mysql>describe blacklist;
5. Verify ip address has been blacklisted
mysql>select * from blacklist where _blacklist='xxx.xxx.xxx.xxx';
6. Delete the entry
mysql>delete from blacklist where _blacklist='xxx.xxx.xxx.xxx';
7. Verify
mysql> select * from blacklist where _blacklist='xxx.xxx.xxx.xxx';
</pre>

The suspected mail server may also be in the helo table.

<pre>
mysql> describe helo ;
1. Check the IP address
mysql> select * from helo where _host='xxx.xxx.xxx.xxx' ;
2. Check the server helo name
mysql> select * from helo where _helo like '%server%' ;
3. Delete the entry
mysql> delete from helo where _host='xxx.xxx.xxx.xxx' ;
mysql> delete from helo where _helo like '%server%' ;
</pre>

====Add ip address to client_reject====

The client_reject list is one of the ways we combat spam. If we receive spam compaints from a mail server or otherwise identitfy a mail server as a source of spam, we can add it to a client_reject list. An email sent from a mail server on the client_reject list is rejected with a notice saying "Relaying denied due to excessive spam". To add an ip address to the list, we use admin-2. Once the list is updated on admin-2, it is pushed out to mta-1 and mta-2. To add an ip address on admin-2:

<pre>
Log into admin-2.nethere.net.
bash-3.00# cd /dist/files/nhmta/nethere/conf/postfix/common
bash-3.00# co -l client_reject
RCS/client_reject,v --> client_reject
bash-3.00# vi client_reject

-Add the ip address to the bottom of the list in the appropriate format. It you want to reject any email from the mail server, you would use:
xxx.xxx.xxx REJECT Relaying denied due to excessive spam
where xxx.xxx.xxx is the /24 subnet of the mail server the spam originated from. Do not place a period after the last octet. For example, to add a /16 subnet you would use:
xxx.xxx REJECT Relaying denied due to excessive spam
The REJECT key word tells postfix to reject the email. If you want to allow email from a /24 subnet, use the following format:
xxx.xxx.xxx OK
This tells postfix to accept email from that subnet.

bash-3.00# ci -u client_reject
RCS/client_reject,v <-- client_reject
new revision: 1.286; previous revision: 1.285
enter log message, terminated with single '.' or end of file:
>> .
done
bash-3.00# make
/nethere/software/nhmta/postfix-2.4.5/sbin/postmap -C . client_reject
mv client_reject.db maps/hash/client_reject.db
bash-3.00# cd /dist/rdist
bash-3.00# gmake nhmta-update
updating host mta-1.mail.nethere.net
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject
updating host mta-2.mail.nethere.net
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject
bash-3.00#
</pre>

====Free mysql disk space on mta-db.mail.nethere.net====

In response to nagios alert "[Nagios] PROBLEM alert - mta-db.mail.nethere.net/Disk mysql is WARNING", to clear space within the mysql database:

1. ssh to mta-db.mail.nethere.net.

2. Log into mysql and execute the following:

[root@mta-db user]# mysql -uroot -p

Enter password: dB@dm1N!

mysql> show databases;

mysql> use policyd;

mysql> show tables;

mysql> describe helo;

mysql> describe throttle;

mysql> describe throttle_from_instance;

mysql> select * from throttle_from_instance limit 10;

mysql> select now();

mysql> select unix_timestamp(now());

mysql> select count(*) from throttle_from_instance where _expire > 1426534697;

mysql> select max(_expire) from throttle_from_instance;

mysql> truncate table throttle_from_instance;

mysql> select count(*) from throttle_from_instance;

mysql> select * from throttle_from_instance;

mysql> show tables;

mysql> optimize table throttle_from_instance;

mysql> show databases;

mysql> use policyd;

mysql> quit

[root@mta-db user]# df -h

== Errors ==

=== Common webmail errors ===

Error:
--cut--
Fatal error: Call to undefined function: applicatio€”¬p() in
/www/webmail.nethere.net/htdocs/x/m/templates/message/navbar.inc on line 7
--cut--

Solution: Restart apache on webmail-1 (apachectl restart)

Error:
--cut--
ERROR There was an error sending your message: unable to add recipient
[webhosting@nethere.com]: Invalid response code received from server
--cut--

Solution: More than likely DNS/domain issue, but check mta-1.nethere.net to be sure:
--cut--
Aug 30 15:08:54 mta-1 nh/smtpd[50501]: NOQUEUE: reject: RCPT from webmail-1.mail.nethere.net[66.63.128.181]: 450 <orders@mulligrins.com>: Sender address rejected: Domain not found; from=<orders@mulligrins.com> to=<webhosting@nethere.com> proto=ESMTP helo=<webmail.nethere.net>
--cut--
Problem: Webmail shows "1-5" messages but inbox appears empty. POP shows messages #1,2,3,4,5... could not be retrieved... server response error cannot open the message file it's gone.

Solution: Ownership issue

1) Use the provisioning tool to determine mailbox location:
a) Click on email account, Engineering
b) Location is the "Storage Path:"

2) Log in to either mail server then change ownership of maildir to mailuser:mailuser
# chown -R mailuser:mailuser /nfs/#/platform/u/s/username

== DEPRECIATED INFORMATION ==

=== Webservers ===

==== Allowing SSI (Server Side Includes) ====

***** NOTE: This has been DEPRECIATED, all servers now allow SSI by default *****

We do not allow EXEC permissions for SSI.

1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>
# co -l www.domain.tld(.common)

2) Add the following lines under the <Directory> directive
--cut--
AddType text/html .shtml
AddHandler server-parsed .shtml
--cut--

3) Edit the "Options" line to be the following:
--cut--
Options Indexes FollowSymLinks IncludesNOEXEC
--cut--

4) Add a DirectoryIndex line above the <Directory> directive with the following:
--cut--
DirectoryIndex index.shtml index.html index.htm home.html home.shtml index.php
--cut--

** In summary, the config should look something similar to this **
--cut--
DocumentRoot "/www/www.domain.tld/htdocs"
DirectoryIndex index.shtml index.html index.htm home.html home.shtml
<Directory "/www/www.domain.tld/htdocs">
AddType text/html .shtml
AddHandler server-parsed .shtml
Options Indexes FollowSymLinks IncludesNOEXEC
Order allow,deny
Allow from all
</Directory>
--cut--

5) Check in Apache config
# ci -u www.domain.tld(.common) < /dev/null

6) Restart Apache
# apachectl configtest
# apachectl restart

More SSI notes can be found here:
http://httpd.apache.org/docs/1.3/howto/ssi.html

= SimplyInternet =

= Z-net =

= Migrating to LAMP Host Servers =

Below are outlines of the steps to migrate website & email hosting from nethere to lamphost configured servers. This is almost certainly incomplete at this stage. The exact steps to follow can vary depending on specifics of the client, so use this as a guide only. Be ready to adjust commands or add steps as needed.

== Edit Nethere DNS ==

ssh user@admin-1.nethere.net

sudo -i

ssh ns1.nethere.net

cd /named/master

co -l example.com

vi example.com

ci -u example.com

rndc reload example.com

Make sure to increment serial number by 1 each time a change is made.

== Migrate Website to nh1.jcihosting.net ==

Set TTL to 300 on ns1.nethere.net

Open up these web admins to get current account/site info:

http://toolbox.nethere.net/ -> Provisioning Tool (search for domain name)

https://secure.johncompanies.com/mgmt/index.html (search for col#####)

https://www.lamphost.com/admin/user/user/create

username: col#####

email: same as listed for col##### account

password: random, secure, won't be shared with client

defaults on rest of page

https://nh1.jcihosting.net:8443/

LAMP Host Users -> Add User -> col#####

Virtual Hosts -> Add Virtual Host

Username: Select same username as above from pulldown

Domain: example.com

Hosting Plan: select plan that matches what's listed in https://secure.johncompanies.com/mgmt/view.html?cid=col#####

Add default DNS: No

Virtual Hosts -> Site Manager (next to domain added)

Site Manager Users -> Add User

Username: can be anything - I typically use Optigold Login from nethere admin

Password: make up something secure and record - this one will go to the user

Email & FTP -> Add Account

Email Account: ftp

Password: make up something secure and record - this one will go to the user

Enable FTP Access: Yes

Optional: Add MySQL database if site needs it (WordPress etc)

ssh to nh1.jcihosting.net and run 'add_mysql.php example.com'

MySQL database info will be emailed to webmaster@lamphost.com (and output on command line - ignore instructions to run additional commands)

ssh nh1

cd /var/www/example.com/

rsync -v --archive --one-file-system --delete --delete-during --rsh=/usr/bin/ssh matt@unixweb-2.nethere.net:/www/www.example.com/htdocs ./

Change 'matt' to your username. Note that I had to add my username to groups httpd & webuser to allow me to copy website files without being root.

chown -R col#####:col##### htdocs

If site has MySQL data find the connection info. For example, WP sites will have the info in htdocs/wp-config.php

mysqldump -h mysqldb-1.webhost.nethere.net -u username --password=password databasename > databse.sql

mysql exampledotcom < database.sql

rm database.sql

If site is ready to be made live on nh1.lamphost.net then update DNS on ns1.nethere.com

Send email to client with details. Search support@jcihosting.com Sent folder for emails with Subject "updated hosting for" for example emails. You will have to adjust as needed (username/passwords, different info based on specific client).

== Migrate Email to nh1.jcihosting.net ==

If migrating website & emails both follow the above steps to transfer the website first.

If migrating emails only the follow the first section of the above steps to set up the virtualhost on nh1.

Set TTL for MX record to 300.

Generate email report:

Log in to http://toolbox.nethere.net/

Select Provisioning Tool and search for the domain

Navigate to the Email section and then Generate Report.

Select these fields for the report:

Mail address

Alias

Forwarding address

Mail storage path

Copy the result of the report and save it as a text file in /tmp/ on nh1.jcihosting.net named example.com.txt (replace domain name)

Make sure the virtualhost has enough email accounts allocated before proceeding. Edit the virtualhosts.virtualhost_settings table directly if need be to increase allocation.

Run this script once and only once:

/usr/local/bin/migrate_mail_accounts_from_nethere.php example.com

That will create all email accounts and aliases.

A file containing the list of email accounts and their new passwords will be created as /tmp/example.com-passwords.txt. Grab a copy of that for sending to the client.

Another file will be created that contains the bash commands to sync emails from the old server. This will be named /tmp/example.com-sync.sh

Copy the file somewhere like the /root/ home dir and add "#!/bin/bash" as the first line and change perms to 750. Now you can run this script to sync all mails from nethere to nh1. Run it immediately to get an initial sync of emails.

When ready to switch email hosting over run the sync script one more time, then update DNS on nh1.nethere.net.

If you'd like to sync again after the move (if there was a delay or you think there may have been incoming emails during the transtion) remove the '--del' option from the sync script and then re-run.

Search support@jcihosting.com Sent folder for messages with Subject "new email hosting for" for examples of emails to use for communicating with the client.