NetHere

NetHere

URL: http://toolbox.nethere.com

Access to servers is restricted to admin-1.nethere.net. Access to admin-1.nethere.net is restricted to our office and nat.johncompanies.com.

JCI Hosting acquired the web-hosting customers of NetHere. (Previously, NetHere had aquired customers from Simply Internet, and Z-net). Customer billing/info are being imported from NetHere to JCI database URL: https://secure.johncompanies.com/mgmt/index.html

Nethere has several tools for management. These are restricted to only allow access from the office, or a few admin's homes.

 cacti [1]
 nagios[2]
 toolbox [3]
 helpdesk [4]
 support FAQ [5]
 knowledge base [6]
 phpmyadmin [7]
 

Virtual Hosts

66.63.129.2 - New HP DL360
fpweb-2.nethere.net
unixweb-6.nethere.net
mta-db.mail.nethere.net
scan-1.mail.nethere.net

66.63.129.4- vmware9.eng
mysqldb-1.webhost.nethere.net

66.63.129.91
lists-1.nethere.net
sb-2.nethere.net
mx-1.nethere.net- spooling server
mailx-1.nethere.net- web server outbound mail server

66.63.129.101 (c.host.nethere.net)
ahi.nethere.net- ldap
mailbox-4
mta-2.mail.nethere.net
relay-2.mail.nethere.net
unixweb-8.nethere.net

66.63.129.102  (vm2.eng.nethere.net)
koi.nethere.net- off- realserver
prov-1.nethere.net- cp.nethere.net, cp.znet.net, cp.simplyweb.net
scribe.nethere.net- prov.nethere.net, scribe.nethere.net
tetra.nethere.net- nethere.com, znet.com, simplyweb.com
web-2.inboxhq.net- helpdesk
webmail-1.nethere.net

66.63.129.103
andromeda.nethere.net (ns2.nethere.net)
nscache-2.nethere.net
nsrbl-2.nethere.net
mailbox-2.nethere.net
scan-2.mail.nethere.net

66.63.129.104
eel.nethere.net- ldap
mta-1.mail.nethere.net
nscache-1.nethere.net
nsrbl-1.nethere.net
phoenix.nethere.net (ns1.nethere.net)
relay-1.mail.nethere.net
unixweb-2.nethere.net

66.63.129.105
home-1.nethere.net
ntweb-11.nethere.net
ntweb-6.nethere.net
cart32.nethere.net
(web-3.schedulecafe.com)

66.63.129.106                  6499MB/16381    18.32GB/268.25 free
admin-1.nethere.net
admin-2.nethere.net
pike.nethere.net- cacti, nagios- off
shark.nethere.net- ldap
ntdb-1.nethere.net
winrestore64

207.167.93.106- vmware6.eng    4156MB/8185MB    45.94GB/267GB free
ntweb-2.nethere.net
unixweb-12.nethere.net (no web sites on server)
mailbox-1.nethere.net

207.167.93.108- vmware8.eng
mailx-2.nethere.net
mx-2.nethere.net

207.167.93.110- vmware10.eng
unixweb-11.nethere.net

Hardware Hosts

Cab 6-08

siron-3 (off)
siron-2
siron-1
106
104
sndg-br-1
sndg-br-2

mailbox-2.nethere.net  (off - virtualized)
mailbox-3.nethere.net  
mailbox-1.nethere.net  (off - virtualized)

scan-1 (OFF)

scan-2  

129.2
102
101

105
103

Backup-2
Backup-1

Cab 5-02

ntweb-3
unixweb-7

unixweb-3
unixweb-2 (now virtualized)
unixweb-10

ntdb-2 or 3?

vmware6.eng

Admin Access

Access to the Admin Systems are protected by firewalls and application filters based on IP addresses.

Main Firewall

The Main firewall is on sndg-cr-1. The main firewall is a Cisco 3750.

IP Access List Editing

1. enable

2. show ip access-lists access-list-name 

   show ip access-list BLOCKED

3. configure terminal

4. ip access-list resequence access-list-name starting-sequence-number increment

5. ip access-list {standard | extended} access-list-name
    
   ip access-list extended BLOCKED 

6. sequence-number permit source source-wildcard
   sequence-number deny protocol source source-wildcard destination destination-wildcard [precedence precedence] [tos tos] [log] [time-range time-range-name] [fragments]
   no sequence-number 
 
7. end

8. show ip access-lists access-list-name

Host Firewall

On the servers, there is often a firewall such as ipfw, ipf, or pf.

Toolbox

From root@admin-1 connect to scribe

 ssh scribe

Stop the scrolling of messages to the screen

 /etc/rc.d/syslogd stop

Add the new IP address to the pf config.

 vi /etc/pf.conf

Reload the firewall rules

 pfctl -F all -f /etc/pf.conf

Add the new IP address to the apache config.

 cd /nethere/conf/apache/conf/vhosts
 co -l prov.nethere.net  scribe.nethere.net.common
 vi prov.nethere.net
 vi scribe.nethere.net.common
 ci -u prov.nethere.net  scribe.nethere.net.common

Reload the apache config

 apachectl restart

Restart the syslog daemon

   /etc/rc.d/syslogd start

Nagios

From root@admin-1 connect to pike

 ssh nagios

Add the new IP address to the apache config.

 cd /nethere/conf/apache/conf/vhosts
 co -l nagios.nethere.net cacti.nethere.net mrtg.nethere.net
 vi nagios.nethere.net cacti.nethere.net mrtg.nethere.net
 ci -u nagios.nethere.net cact.nethere.net mrtg.nethere.net

Reload the apache config

 apachectl restart

Application Filter

In the applications, there is often a restriction in the host application configuration file.

 vi /usr/local/etc/apache/conf/vhosts/<host>.conf

Restart the application

 apachect restart

NetHere/zNET/simplyweb Cert replacement

If you wish to use the old CSR, Global Sign keeps the old CSR, so all you have to replace is the Cert.

To generate a new csr for each

openssl req -nodes -newkey rsa:2048 -keyout wild.nethere.key   -out wild.nethere.csr
openssl req -nodes -newkey rsa:2048 -keyout wild.simplyweb.key -out wild.simplyweb.csr
openssl req -nodes -newkey rsa:2048 -keyout wild.znet.key      -out wild.znet.csr

Get a wild card cert for each company (nethere.net, simplyweb.net. znet.net)
https://www.globalsign.com/en/

Update certs, keys, and/or pem files on the following servers.

webmail-1.mail (webmail for nh, si, and zn)          /nethere/conf/apache/pki/ssl.crt
prov-1.mail (cp for nh, si, and zn)                  /nethere/conf/apache/pki/ssl.crt
mta-1 and mta-2 (smtp)                       admin-2:/dist/files/nhmta/nethere/conf/postfix/nh/ssl
                                                     /nethere/conf/postfix/nh/ssl
                                                     /nethere/conf/postfix/si/ssl
                                                     /nethere/conf/postfix/zn/ssl
mailbox-1, mailbox-2, and mailbox-3 (pop3 and imap) (courier-imap)
                                                     /nethere/conf/courier-imap/nh
                                                     /nethere/conf/courier-imap/si
                                                     /nethere/conf/courier-imap/zn
              (smtp)

Provisioning

Webservers

OS: FreeBSD 4.11-RELEASE-p10 #23

General

unixweb-1 through 8
  
All have the same configuration except for the following exceptions:

unixweb-4: ZN Front Page server (depreciated)
unixweb-5: ZN Front Page, Miva Merchant server
unixweb-6: SI shared counter server (for SI sites): /www/lucy.inetworld.net/htdocs/cgi-bin/Count.cgi 

Server Software Installed

Apache
	# httpd -v
	Server version: Apache/2.0.55
	Server built:   Apr  5 2006 17:04:01
PHP
	# /usr/local/bin/php4 -v
	PHP 4.4.2 (cgi-fcgi) (built: Mar  2 2006 09:31:57)

Miva Merchant (unixweb-5)
MySQL
	# mysqladmin version -u root -p
	Server version          4.0.25

Apache configurations are in the following directories:

/nethere/conf/apache/conf/nvhosts  (name based hosting); naming convention is www.domain.tld
/nethere/conf/apache/conf/vhosts (ip based,SSL); naming convention is www.domain.tld (IP information) and www.domain.tld.common (general site information)

NetHere specific scripts are located in:
/nethere/sbin

Checking Server/site status via mod_status apache module:

# apachectl stop
# /usr/local/etc/rc.d/httpd.sh start-status

URL: http://unixweb-#/status (replace # with the unixeweb number)
** NOTE: After viewing status, do the following, otherwise other users can also view status by using .htaccess.
# apachectl stop
# apachectl startssl

Provisioning

Provisioning new sites:

• • • All website provisioning is done via NH script (webadd) on sawfish (admin-1.nethere.net) ***

1. /nethere/sbin/webadd -h for usage

Note: We've discontinued new website provisioning on all servers except for unixweb-6, unless otherwise noted for domain

FTP Password Adjustment

Unix Server

To change an FTP password:

On the customers server as root do

passwd <username>

(The password prompt will not echo entry)

Windows Server

To change an FTP password:

Find the users login (ie aa5014) in the provisioning tool.
Use Remote Desktop to get into the server.
On Desktop find FTP application Icon.
Use that tool to update the user's password.

Domain Aliasing

For domain aliasing:

1) Edit the /named/named.master on phoenix (ns1.nethere.net)

 a) Checkout the file; open with vi
    # co -l named.master
    # vi named.master
 b) find the domain that's serving as the master domain, then, following the general format of the file to add the domain aliases under the master domain entry, using the master domain zone file
    i.e.  
    --cut--
    zone "domain.tld" { type master; file "master/domain.tld"; };
    zone "aliased_domain.tld" { type master; file "master/master_domain.tld"; }; 
    --cut--
 c) save the file, then check-in:
    # ci -u named.master < /dev/null
 d) update the zone records, using the makefile in /named; check for errors
    # make new-zone
    # tail /var/log/named
 **Note: some domains are IP based virtually hosted (i.e. SSL certificate). This requires a modification to the standard aliasing procedure. If this is the case, do the following
 e) Create a new zone file called "domain.tld-alias" in /named/master, copy current domain.tld zone file to the domain.tld-alias file, and adjust the "website" records, removing whatever IP based information is there, and replacing with the appropriate $INCLUDE name-based host template.
    e.g.
    tropicalshade.net:
    --cut--
    ;; ntweb-4
    @               IN      A       66.63.136.4
    www             IN      A       66.63.136.4
    --cut--
    tropicalshade.net-alias
    --cut--
    ;; website
    $INCLUDE master/nvhost.ntweb-4
    --cut--
 f) Adjust the named.master zone file in /named accordingly:
    i.e.  
    --cut--
    zone "aliased_domain.tld" { type master; file "master/master_domain.tld-alias"; }; 
    --cut--

2) Edit the Apache configuration for the domain on the web server the master domain is hosted on

  a) Script is /nethere/sbin/webalias
     # /nethere/sbin/webalias -h (for usage)
     e.g. # /nethere/sbin/webalias -d www.originaldomain.tld -a www.domainalias.tld

CGI Scripting

For security, we've implemented suexec on all sites. All CGI *must* be placed in the cgi-bin/ (and/or cgibin/ directory for unixweb-4, 5) directory

CGI permissions break down as follows:

Ownership: user=username, group=webuser Permission: cgi-bin/ = 755; files = 755

suexec errors can be found here: /www/default/logs/suexec_log

cgi errors for a domain can be found in the main apache error logs for the domain: /www/www.domain.tld/logs/error_log

More information on CGI scripting can be found here: http://httpd.apache.org/docs/1.3/howto/cgi.html

Formmail Provisioning

Formmail is provisioned via NH script (newformmail) on server domain is hosted on.

1. /nethere/sbin/newformmail -h for usage

Usage: newformmail [-h] [-d domain] [-i ip_address] [-e "email1 email2"]

Disabling a site

To disable/enable a website, there is a script (webstatus) located on each webserver: /nethere/sbin/webstatus -h for usage

1) Disable:

  # /nethere/sbin/webstatus -r -d -s www.domain.tld

2) Enable:

  # /nethere/sbin/webstatus -r -e -s www.domain.tld

Stats Provisioning

We utilize webalizer for statistics.

Stats are provisioned via NH scripts on server domain is hosted on.

1) Provision stats for the domain (webaddstats_unix):

1. /nethere/sbin/webaddstats_unix -d www.domain.tld

2) Run stats for the domain (runstats_unix):

1. /nethere/sbin/runstats_unix www.domain.tld

More information on Webalizer can be found here: http://www.mrunix.net/webalizer README: ftp://ftp.mrunix.net/pub/webalizer/README

Removing a site

1) Site removal is primarily done via a NH script (webdel) on the webserver the domain is hosted on:

  # /nethere/sbin/webdel -h for usage
  # /nethere/sbin/webdel -r -d www.tldomain.com 

2) Update DNS accordingly (ns1.nethere.net)

  a) Checkout /named/named.master
     # co -l named.master
  b) Remove line containing domain
  c) Checkin /named/named.master
     # ci -u named.master < /dev/null
  d) Update zones
     # make new-zone
  e) Move zone files from /named/master to /named/archive
     # mv domain

starter removal

All starter sites on home-1.nethere.net

1) Removal primarily done via a NH script (webdel_home) on home-1

  # nethere/sbin/webdel_home -h for usage
  i.e. webdel_home -p zn -s startername  <== for removing znet

Disk Quota increases

Quotas are increased by using the "edquota" command:

1. edquota username

Note that quotas are in KB, so a conversion is necessary. Also note that the "soft" quota is 1 MB less than the "hard" quota.

The calculations are as follows: "soft" = ( quota(in MB) - 1 ) x 1024 "hard" = quota(in MB) x 1024 e.g. for a 500 MB quota: soft => (500 - 1) x 1024 = 510976 hard => 500 x 1024 = 512000

To check a quota:

1. quota -v username

You can also use the prototype users:

1. edquota -p quota100 username (100 MB)
2. edquota -p quota500 username (500 MB)
3. edquota -p quota1000 username (1000 MB)

Granting of shell - Enterprise packages

Shell is granted by the "chsh" command. By default, C shell (csh) is used:

1. chsh -s shell_needed username

e.g. # chsh -s csh username

• Note: customers must provide us with the static IP address(es) they will be connecting from. The IP address(es) must be added to the configuration on admin-1.

1) Modify the corresponding rules files on admin-1: /dist/files/nhweb/etc/ipf.rules.fxp0 , .em0 (RCS controlled) 2) Push updates to servers

1. cd /dist/rdist
2. gmake nhweb
3. gmake nhweb-update

3) Reload the rules on the corresponding server that needs access to

1. ipf -Fa -f /etc/ipf.rules

SSL certificate installs

1) Whois the site for information on Registrant, email - to be used for SSL generation

1. whois domain.tld

2) Create SSL self signed certificate, get CSR for customer to sign

 a) SSL generation is done by script: /nethere/conf/apache/pki/newsslcert.sh 
 b) Copy CSR for customer

3) Create/update Apache configuration files via NH script (webadd_ssl)

 # /nethere/sbin/webadd_ssl -h for usage
 a) webadd_ssl [-h] [-d domain] [-s ssl_domain] [-n]
 # webadd_ssl -d www.domain.tld -s www.domain.tld 

4) Update DNS zone with new IP address - done on ns1.nethere.net

 a) Check out DNS zone file in /named/master
    # co -l domain.tld
 b) Adjust A records:
  --cut--
  ;; unixweb-##  << enter the unixweb server number for ease of ID
  @		IN	A	vhost_ip_address
  www	IN	A	vhost_ip_address
  --cut--	
 c) Check in DNS zone file
    # ci -u domain.tld < /dev/null
 d) Reload zone file
    # rndc reload domain.tld
 e) check /var/log/named for errors

5) Update DNS PTR record for IP address - done on ns1.nethere.net

 Note: Assuming IP address a.b.c.d
 a) Check out in-addr.arpa zone for IP address in /named/master
    # co -l a.b.c 
 b) Follow format for PTR records   --cut--
  d	IN	PTR	www.domain.tld.
  --cut--
 c) Check in PTR zone file     # ci -u a.b.c < /dev/null
 d) Reload PTR zone 
    # rndc reload c.b.a.in-addr.arpa
 e) check /var/log/named for errors

Updating SSL certificate

1) cd to /nethere/conf/apache/pki/ssl.crt on server site is hosted on

2) Check out (RCS) www.domain.tld.crt file

  # co -l www.domain.tld.crt

3) Edit the file, remove old certificate, paste in new certificate

4) Check in (RCS) the www.domain.tld.crt file

  # ci -u www.domain.tld.crt < /dev/null

5) Check, restart Apache

  # apachectl configtest
  # apachectl stop
  # apachectl startssl

6) Verify httpd started:

  # ps auxw | grep httpd
 a) If no processes, will need to revert back to old SSL cert, and restart apache. Check logs for errors
    # view /www/default/logs/ssl_engine.log
 b) Check for "Unable to configure RSA server private key" and "key values mismatch" entries - this means a bad SSL certificate

Password protection

Http (simple) password protection is governed by the Apache configuration for the domain

1) Create userdb, users file in the domain root directory (/www/www.domain.tld):

  # mkdir userdb
  # cd userdb
  # htpasswd -bc users username password

2) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>

  # co -l www.domain.tld(.common)  

3) Edit Apache configuration, add the following lines within the VirtualHost container --cut-- <Directory "/www/www.domain.tld/dir_to_be_protected">

   AuthType Basic
   AuthName "www.domain.tld/dir_to_be_protected authentication"
   AuthUserFile /www/www.domain.tld/userdb/users 
   <Limit GET POST>
     require valid-user
   </Limit>
 </Directory>

--cut--

4) Check in Apache config

  # ci -u www.domain.tld(.common) < /dev/null  

5) Restart Apache

  # apachectl configtest
  # apachectl restart

More on http (simple) password protection can be found here: http://httpd.apache.org/docs/1.3/howto/auth.html#basic

.htaccess

Used if customers want control of certain Apache directives (i.e. Authentication, etc.)

1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>

  # co -l www.domain.tld(.common)  

2) Add the AllowOverride directive in the <Directory> section, under the PHP FCGIWrapper

  i.e.
 <Directory "/www/www.domain.tld/htdocs">
   FCGIWrapper /www/www.domain.tld/htdocs/cgi-bin/php4 .php
   AllowOverride AuthConfig FileInfo Indexes Limit
 </Directory>

3) Check in Apache config

  # ci -u www.domain.tld(.common) < /dev/null  

4) Restart Apache

  # apachectl configtest
  # apachectl restart

More info on Allow Override can be found here: http://httpd.apache.org/docs/1.3/mod/core.html#allowoverride

Domain re-provisioning

1. 1. NOTE: Following is for domain being provisioned on same server (i.e. just being renamed)

1) Adjust DNS - ns1.nethere.net

 a) Rename the DNS zone file to the new domain, remove the old DNS zone files from /named/master and /named/master/RCS
 b) Check in the new DNS zone file
 c) Edit /named/named.master - replace the old domain with the new one, alias as necessary
 d) Reload DNS zones - make new-zone in /named

2) Adjust Apache config - server domain is hosted on

 a) Find current config file(s) in /nethere/conf/apache/conf/<vhosts,nvhosts>
 b) Replace old domain name entries with new domain name, alias as necessary
 c) Save as new_domain.tld
 d) Remove old_domain.tld(.common), RCS/old_domain.tld(.common)
 e) Check in new_domain.tld via RCS:

# ci -u new_domain.tld < /dev/null

 f) Check out Apache include configuration file via RCS /nethere/conf/apache/conf/<nvhosts.conf,vhosts.conf> 
 g) Edit nvhosts.conf or vhosts.conf, replace old_domain.tld entries with new_domain.tld
 h) Check in via RCS /nethere/conf/apche/conf/<nvhosts.conf,vhosts.conf>

3) Rename directory for new domain

  # cd /www
  # mv www.old_domain.tld www.new_domain.tld

4) Adjust PHP stub files/configuration

 a) Adjust php.ini file in /www/www.domain.tld/(php4,php5)
 b) Adjust PHP stub files in /www/www.domain.tld/htdocs/cgi-bin/(php4,php5) 
 *note: need to chflags to "noschg" for /www/www.domain.tld/htdocs/cgi-bin/(php4,php5) before being able to update the stub files
	 after adjusting, be sure to chflags schg /www/www.domain.tld/htdocs/cgi-bin/(php4,php5)

5) Edit password file (vipw), replace old_domain.tld entries with new_domain.tld

  # vipw
  

6) Restart Apache

  # apachectl configtest
  # apachectl restart

1. 1. NOTE: Following is for domain being re-provisioned on new server

1) Run NH script "webadd" on sawfish to provision domain on new server, *DO NOT* reload DNS, use same user/pass as before. 2) After customer has uploaded site to new server and gives the ok do the following:

 a) Update DNS zone for domain.tld; reload zone for domain.tld
 b) Wait 48 hours, then remove the site off the old server
    # /nethere/sbin/webdel

FTP space provisioning

Generally FTP sites are provisioned on the server that hosts the main website. In the case of NT based FTP sites, we usually provision them on the server with the most space available.

1) Provision site on server

 a) Done via NH script (webadd_ftp): webadd_ftp [-h] [-d domain] [-u username] [-p password] [-n]
    # /nethere/sbin/webadd_ftp -h  <- for usage
    EX: for ftp.domain.tld:
    # /nethere/sbin/webadd_ftp -d ftp.domain.tld -u username -p password
    (leaving off the -n will restart the proftpd process)
 b) Note: The host IP address will be given when the provisioning is completed, use that ip (a.b.c.d) for DNS entries
 c) Note: If this is an existing customer on the server, you'll need to increase the quota manually by 100 MB for the customer, see the section on quota increases for more info.

2) Adjust DNS for domain on ns1.nethere.net

 a) Checkout zone for domain
 b) Add ftp host entry for domain, adjust serial
    EX for domain.tld:

--cut-- ftp IN A a.b.c.d --cut--

 c) Check in zone for domain
 d) Reload zone
    # rndc reload domain.tld

3) Adjust PTR record for domain, adjust serial

 a) Checkout zone for a.b.c
 b) Add record for domnain:

--cut-- d IN PTR ftp.domain.tld. --cut--

 c) Check in zone for a.b.c
 d) Reload zone
    # rndc reload c.b.a.in-addr.arpa

4) Check for DNS errors

 a) tail /var/log/namedb

SiteBuilder provisioning

unixweb-7.nethere.net

• Must re-provision site on unixweb-7.nethere.net (if not already done)

1) Log in to SB admin:

  http://sitebuilder.nethere.net/admin
 a) username: root

2) Add site to SB config

 a) Click on Site Management -> Add regular
   * Alias is website username: i.e. aa####
   * Check the "Active" box  
   * Plan is "BasePlan"  
   * Password same as website
 b) Click on "Publish Properties
   * Check "Allow publishing"
   * Site host name: www.domain.tld
   * FTP host: unixweb-7.nethere.net
   * FTP login/password: same as site user/pass
   * FTP working directory: leave blank
 c) Click Apply

Name Servers

OS: FreeBSD

General

Nethere DNS has been migrated to PowerDNS on ganeti virtuals with a web admin here: https://nhdns.jcihosting.com/ Old instructions follow:

ns1.nethere.net - Primary name servers for DNS zone records

/named - contains the files that have all DNS domain zone entries (named.master, named.slave, named.acl) plus Makefile for distributing DNS records
/named/master - contains all the domain zone files for which we are authorative for, as well as IP address (PTR records)

ns2.nethere.net - Secondary (slave) name server for DNS zone records

nsrbl-1.nethere.net - RBL (Realtime Blackhole List) DNS server

/named/rbldns/cache - contains the files for domains that we specifically allow or deny

nscache-1,2 - caching name servers

Adding DNS website entries

1) Create a DNS zone file for domain.tld via NH script (zoneadd_vhost)

  # /nethere/sbin/zoneadd_vhost -h  (for usage)

Adding DNS IP entries

1) Create the forward and reverse DNS records via NH script (zone_generate) for a netblock, will create /tmp/customer.forward and /tmp/customer.reverse files to be read

 a) /nethere/sbin/zone_generate -h for usage
   i.e. for netblock a.b.c, starting IP d, ending IP z
    # zone_generate -n a.b.c -b d -e z -p customer

2) Checkout, edit the IP in-addr.arpa zone file, reload the zone

 a) # co -l a.b.c
 b) Seach for the nearest netblock area for the domain, follow format for customer info, read in the /tmp/abbrev.rdns file accordingly, increase Serial for zone in YYYYMMDD## format
    e.g.

--cut--

66.63.152.232/30 (255.255.255.252)

Description

First Choice Home Improvement

Contact

Shannon Hill <firstchoicehi@hotmail.com>, (858) 277-5351

Location

AR-1, Serial3/0/18:0

232 IN PTR firstchoice-net.access.nethere.net. 233 IN PTR firstchoice-gw.access.nethere.net. 234 IN PTR firstchoice-2.access.nethere.net. 235 IN PTR firstchoice-bcast.access.nethere.net. --cut--

 c) # ci -u a.b.c < /dev/null
 d) # rndc reload c.b.a.in-addr.arpa
 e) verify loading of zone: # tail /var/log/named

3) Checkout, edit the forward DNS zone file, reload the zone

 a) # co -l access.nethere.net
 b) Search for the nearest neblock area for the domain, follow format for customer, read in the /tmp/abbrev.fdns file accordingly, increase Serial for zone in YYYYMMDD## format
    e.g.

--cut--

66.63.152.232/30 (255.255.255.252)

firstchoice-net IN A 66.63.152.232 firstchoice-gw IN A 66.63.152.233 firstchoice-2 IN A 66.63.152.234 firstchoice-bcast IN A 66.63.152.235 --cut--

 c) # ci -u access.nethere.net < /dev/null
 d) # rndc reload access.nethere.net 
 e) verify loading of zone: # tail /var/log/named

Unblocking RBLd IP addresses

There are two primary reasons why we add customers to the allow relay list: a) They have a static IP address (i.e. DSL, T1) and wish to use our mail servers b) They've been blocked by one of our subscribed blackhole lists, however, have patched their machine and are no longer open to relay.

To do this on nsrbl-1.nethere.net:

1) Checkout the allow.relays.nethere.net file located in /named/rbldns/cache

2) Edit the file, and add the IP address in the following format: --cut-- a.b.c.d YYYYMMDD hostname reason for listing --cut--

You can also add subnets via '/' notation for relay --cut-- a.b.c.d/28 YYYYMMDD hostname reason for listing --cut--

3) Check in the file

4) Updates to the rbldns zone are done automatically on the hour, every hour, so no need to do anything else.

Note:

For "permanent" (i.e. customer static IP addresses), add the IP address in the "## permanent allowed relay (i.e. customer w/ static IP)" section For "temporary" (i.e. blackhole listed IP addresses), add the IP address in the "## temporary" section

Also, for temporary IP addresses, we need to send the note to the requester detailing the following: a) If the IP address relays Spam/UCE/Viruses through us, it is to be removed permanently. b) The customer must follow the steps listed on the blackhole list the IP was listed on to get removed.

Flushing DNS cache for a domain

Due usually to a bad zone or excessively long TTL for a domain, the cache for it will need to be flushed.

On nscache-1:

1. /usr/local/sbin/rndc flushname domain.tld

If cache is still corrupted (i.e. zone lookups either fail or are incorrect), need to stop and restart the caching server:

1. /etc/init.d/local.named stop
2. /etc/init.d/local.named start

SiteBuilder

http://sitebuilder.nethere.net/ -URL used to test sitebuilder

http://sitebuilder.nethere.net/Login -URL for control panel.

This is where the customer also logs in to manage their web site and also where you log in to administer sitebuilder. Sitebuilder is hosted on sb-2.nethere.net. The site is designed/built on this server and published to unixweb-7.nethere.net. This is the only server that can host a sitebuilder web site. The admin log in to manage sitebuilder is: Username: admin Password: N3tH3r31!

Cart32

cart32 information:

Cart32 is hosted on ntweb-6.nethere.net. That is where all of the configuration files are located. The location of the ini file is D:\websites\Cart32cgi/cart32.ini. This is where ip restrictions to admin panel are set, password can be reset for admin, time limit restriction reset, etc.

There are four customers that still use cart32:

https://www.cart.simplyweb.net/lab400/cart/c32web.exe
https://www.cart.simplyweb.net/nutragenics/cart/c32web.exe
https://www.cart.simplyweb.net/retrogen/cart/c32web.exe
https://www.cart.simplyweb.net/stonesculptorssupplies/cart/c32web.exe

The client codes are lab400, nutragenics, retrogen, and stonesculptorssupplies. You can reset their passwords through the admin panel.

To administer cart32:

Control Panel: https://www.cart.simplyweb.net/cart/c32web.exe/Admin Username: administrator Password: N3tH3r31! Cart Admin Password: N3tH3r31!

Webmail

Removing webmail filters

All webmail filters are stored in the MySQL database on webmail-1.

Simple SQL commands:

• Choosing a DB

mysql> use db_name;

• Showing tables in a DB (must be using a DB)

mysql> show tables;

• Describing a table's fields

mysql> describe table_name;

For NetHere, the database is horde_nh. For Simply, the database is horde_si. For zNET, the database is horde_zn.

1) Log to MySQL as root, using the normal root password

 a) # mysql -u root -p

2) Select the appropriate database

 a) mysql> use horde_nh

3) Delete webmail filters.

 a) mysql> delete from horde_prefs where pref_uid='<username>' and pref_name='filters';
    Replace <username> with the users login.

Removing Address book entries

1) Log in 2) Select appropriate db 3) Find object_id, delete

  a) mysql> select object_id from turba_objects where object_email='email@domain.tld';
  b) mysql> delete from turba_objects where object_id='object_id_obtained_previously';

Repairing webmail address book

1) Export (via webmail, if possible) the address book to a csv file.

2) Delete the Turba objects in the mySQL database in the turba_objects table that is assigned to the customer.

 mysql> delete from turba_objects where owner_id='username';

3) Import the saved address book via webmail.

Changing webmail From Information

1) Login as the customer (http://webmail.nethere.net)

2) Click on Options

3) Click on Personal Information

4) Select either default identity or a new one and click on Edit Your Identities

MySQL

All unixweb-## boxes have MySQL processes running on them. DB provisioning is done on the same server as which hosts the domain. For Windows 2000 MySQL provisioning, all DBs are placed on ntdb-2.nethere.net

MySQL DB provisioning

Use https://phpmyadmin.nethere.net User: root PW: <root db pw>

1) Select the server that the DB needs to be hosted on, typically the same server that the website is hosted on. 2) Create the DB

 a) Under "MySQL" - "Create new database" - the DB name is typically the website username; click "Create"

3) Create the User, set permissions for user on DB

 a) Click "Home" (upper left), then click "Privileges" (Under "MySQL")
 b) Click "Add a new User" make sure "Any host" selected, "User name" is typically the website username, "Password" is typically the website password. !LEAVE "Global Privileges" settings deselected! Click "Go" (bottom right corner) when finished.
 c) Under "Database-specific privileges" be sure to add the user database
 d) Select DB, then for "Privileges:" make sure the following are checked:
    Select, Insert, Update, Delete, Create, Alter, Index, Drop, Create Temporary Tables, Lock Tables, References
 

List Server

lists-1.nethere.net

Mail List provisioning

1. Go to the /mailman/bin directory: # cd /mailman/bin

2. Create the list on lists-1: # ./newlist -q \ listname@lists.domain.com \ mailman-owner@lists.domain.com \ password

3. Configure the list using default settings: # ./config_list -i /mailman/data/defaultlist.cfg listname

• • • IF A NEW DOMAIN FOLLOW INSTRUCTIONS BELOW ***

NOTE: All files in /etc/mail are controlled by RCS

4. Go to the /etc/mail directory: # cd /etc/mail

5. Add the list domain to the mailertable file: lists.domain.com mailman:lists.domain.com

6. Add the list domain to the relay-domains file.

7. Add the list domain to the virtuserdomain file.

8. Add the following entries to the virtusertable file: mailman@lists.domain.com <customer_email_address> mailman-owner@lists.domain.com mailman@lists.domain.com

9. Run make to rebuild all files: # make

10. Run make to restart sendmail: # make restart

Backup Server

backup-1 -> odd numberd unix backup-2 -> even numbered unix

On Saturdays, we do a level 0 dump the rest are differential

Restoration fees are basically $25 per day restored for email, $150 for website from cancelled archive, $25 a day for website

Restoration is based on customer request. In order to restore a site/mailbox fully, you must start restoration from the previous level 0 backup and continue until the day after the day requested, since backups are performed in the morning.

General rule is to leave the gzipped file available for 24 hours, after that remove it.

Site Restoral

If the site was removed by the web_del script, then so long as it is within 30 days since removal, the gzipped site will be available within /www/archive/www.domain.tld-date_removed.tar.gz Extract with tar:

1. tar -zxvpf www.domain.tld-date_removed.tar.gz

If the site needs to be restored from backup, here are the steps

1) To determine where backups are stored, go to /nethere/conf/backup

2) grep for the server within the directory to determine the backup directory used i.e.

1. grep unixweb-1 *

4) Go to /backup1/unix/unixweb-1.nethere.net

 a) you'll see a bunch of directories, named <date>-<dump_level>

5) Since the backups happen in the morning, typically you'll need to go to the day before, restore

  e.g. for 11/10
 a) cd 20041109-3
 b) restore -if and whatever the filesystem is named
    (for interactive: add files/dirs, extract, then use 1 for volume #)
    i.e.
    restore> add <path_to_dir/files>
    restore> extract
    (for volume #: 1)
    (Set owner: y)

6) After you've restored the files, just use scp as root on sawfish to copy them off of backup-# and then onto unixweb-#

 a) (on sawfish - two step process)
    # scp backup-1:/tmp/<restored_file> /tmp
    # scp /tmp/<restored_files> unixweb-1:/tmp

7) Cleanup any restored files off of backup-# server and sawfish

With the newly restored files, following are the steps necessary to restore a site:

1) Restore www.domain.tld directory to /www

  # mv /www/archive/www.domain.tld /www

2) Restore configuration files

 a) Apache: mv /www/archive/nethere/conf/apache/(n)vhosts/www.domain.tld /nethere/conf/apache/conf/(n)vhosts/
 b) Webalizer: mv /www/archive/nethere/conf/webalizer/unix/www.domain.tld /nethere/conf/webalizer/unix

3) Add user to /etc/master.passwd:

 a) vipw
    (go to end of file, read in the master.passwd file from /www/www.domain.tld)
    :r /www/www.domain.tld/master.passwd
 b) Remove /www/www.domain.tld/master.passwd

4) Change flags on cgi-bin/ directories:

  # chflags sunlnk /www/www.domain.tld/htdocs/cgi-bin/
  # chflags schg /www/www.domain.tld/htdocs/cgi-bin/php*

5) Add configuration file entry to nvhosts/vhosts.conf

  # co -l (n)vhosts.conf
  # vi (n)vhosts.conf
  # ci -u (n)vhosts.conf

6) Check apache, restart; verify apache

  # apachectl configtest
  # apachectl graceful
  # 

7) Cleanup remaining restored files

  # rm -r /www/archive/www
  # rm -r /www/archive/nethere

8) Enter zone into DNS on ns1.nethere.net

 a) zone file
    # mv /named/archive/domain.tld /named/master
    # mv /named/archive/domain.tld,v /named/master/RCS
 b) named.master file 
    # co -l /named/named.master
    (add zone)
    # ci -u /named/named.master
    # cd /named; make new-zone
    # tail /var/log/named   (look for errors)

(12:40:47 PM) Henry Chan: restore is now available that is compatible with the 4.4bsd format (12:41:04 PM) Henry Chan: to restore, use the following command: restore -c -i -f path_to_archive (12:41:08 PM) Henry Chan: the "-c" is what does it (12:41:29 PM) Henry Chan: (only applies to backup-1-new... doesn't work on backup-2 or backup-4)

Email Restoral

Storage path prefixes:
	/nfs/1 is sndg-netapp-1 (on backup-4 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)
	/nfs/2 is sndg-netapp-2 (on backup-3 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)
	/nfs/3 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)
        /nfs/4 is sndg-netapp-1 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)
        /nfs/5 is sndg-netapp-2 (on backup-1 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)
        /nfs/6 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)

Looking up a mailbox's Storage Path:
	You will need to know which directory and NFS server the customer's mail is 
stored on. Use the Provisioning Tool to get the Storage Path (find the customer's mail
account, then click on the Engineering sub-tab under the Email tab). You should end up
with something like "/nfs/3/nh/h/t/htchan/Maildir". Match the prefix of the storage
directory with an NFS server (see prefixes above).



1) If restoring a recently deleted mail account:
	a) Look for the archive on mailbox-1 in /nfs/archive/mail/{platform}/{username}.{date}-{PID}.tar.gz. 
           If it doesn't exist, it's been too long and the only way to get email back is to restore from backup.
	b) Make sure the account is re-created in Provisioning Tool and look up the storage path 
           (see "Looking up a mailbox's Storage Path" above).
	c) Extract the archive to a temporary directory:
		Sample command:
			cd /tmp; tar xzvpf /nfs/archive/mail/nh/zella.20090902-7508.tar.gz
		Sample output:
			nfs/2/nh/z/e/zella/
			nfs/2/nh/z/e/zella/Maildir/
			nfs/2/nh/z/e/zella/Maildir/tmp/
			nfs/2/nh/z/e/zella/Maildir/new/
			nfs/2/nh/z/e/zella/Maildir/cur/
			nfs/2/nh/z/e/zella/Maildir/maildirsize
			nfs/2/nh/z/e/zella/Maildir/.Trash/
			nfs/2/nh/z/e/zella/Maildir/.Trash/tmp/
			nfs/2/nh/z/e/zella/Maildir/.Trash/new/
			nfs/2/nh/z/e/zella/Maildir/.Trash/cur/
			nfs/2/nh/z/e/zella/Maildir/.Trash/maildirfolder
			nfs/2/nh/z/e/zella/Maildir/.Drafts/
			nfs/2/nh/z/e/zella/Maildir/.Drafts/tmp/
			nfs/2/nh/z/e/zella/Maildir/.Drafts/new/
			nfs/2/nh/z/e/zella/Maildir/.Drafts/cur/
			nfs/2/nh/z/e/zella/Maildir/.Drafts/maildirfolder
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/tmp/
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/new/
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/cur/
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/maildirfolder
			nfs/2/nh/z/e/zella/Maildir/courierpop3dsizelist
		Note:
			In the above output, notice that the Maildir is located in "nfs/2/nh/z/e/zella", it will be used in the next step.
	d) Copy the Maildir files to the new mailbox storage path using tar (tar does a better job at preserving 
           things like symbolic links and permissions than cp or mv):
		Sample command:
			cd /tmp/nfs/2/nh/z/e/zella; tar cf - Maildir | (cd /nfs/3/nh/z/e/zella; tar xvpf -)
		Output:
			You should list the files as it is copied. Basically, similar to what you 
                        saw in step "c" above but without the "nfs/2/nh/z/e/zella" prefix.
		Note:
			The command format is basically:
				cd /tmp/{restored-maildir-path-see-note-in-step-c}; tar cf - Maildir | (cd {new-storage-path-without-Maildir}; tar xvpf -)
	e) Clean up temporary directory:
		Sample command:
			cd /tmp; rm -rf nfs

2) If restoring a zfs mailbox (from the last week) to a particular date's backup:
        a) login to the correct sndg-netapp-[1/2/3]-new
        b) get the files from the correct zfs snapshot
           Sample commands:
             cd /tank0/mail/.zfs/snapshot/20180215-0/nh/c/h/christineat
             ls -l
           cd to the desired directory and copy the files over to the users mailbox
             cp -p /tank0/mail/.zfs/20180215-0/nh/c/h/christineat/Maildir/cur /tank0/mail/nh/c/h/christineat/Maildir/cur

3) If restoring a mailbox to a particular date's backup:
	a) Look up the storage path to the customer's mailbox and determine which backup server the 
           dump file is on (see "Looking up a mailbox's Storage Path" and "Storage path prefixes" above).
	b) Restore the level 0 and any incremental backups (in order) to /tmp on the backup server:
		Sample commands:
			cd /tmp
			restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090912-0/mail.dump
			chflags -R 0 /tmp
			restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090913-1/mail.dump
			chflags -R 0 /tmp
				...skipped repetitive stuff here...
			restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090918-6/mail.dump
			chflags -R 0 /tmp
		Note:
			After each restore, we need to recursively remove all flags from /tmp to eliminate 
                       the immutable flag that gets set on the files (this happens only on NetApp dumps... who knows why).
	c) Create a new tar of the Maildir directory so that it can be copied to mailbox-1 for further processing:
		Sample command:
			cd /tmp/nh/z/e/zella; tar cf /tmp/archive.tar Maildir
	d) Clean up /tmp:
		Sample command:
			rm -rf /tmp/nh
		Note:
			Depending on the platform, it might be /tmp/nh, /tmp/si, or /tmp/zn.
	e) Copy the /tmp/archive.tar file to /tmp on mailbox-1.
	f) On mailbox-1, extract the Maildir archive on top of what they already have:
		Sample command:
			cd /nfs/2/nh/z/e/zella; tar xvpf /tmp/archive.tar
	g) Remove /tmp/archive.tar.

Beginning of old instructions

Use the same techniques as site restoration, with the following exceptions:

1) Since everything is in maildir format, you have to reassmble things in /tmp/<username>, tar it up, and then restore it on any of the mailbox machines

  a) using the following syntax: 
     # tar -zcpf /tmp/username.tgz yyyymmdd-#/platform yyyymmdd-#/platform yyyymmdd-#/platform
     i.e.
     # tar -zcpf /tmp/username.tgz 20050915-5/nh 20050914-4/nh 20050910-0/nh

2) In regards to mail spools, you'll need to use the chflags command to adjust the flags on the files. For whatever reason, they are stored with a system immutable flag

  a) Do chflags -R noschg  <dir> on the restored directory before copying the files with sawfish otherwise, the files cannot be deleted:
     i.e. 
     # chflags -R noschg /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/
  b) After tar/gzipping, remove the restored directories
     i.e
     # rm -r /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/nh

3) Can restore quickly with script on mailbox-1 (or mailbox-2) using the NH script "restoremail"

  a) # /nethere/sbin/restoremail -h for usage
     i.e. 
     # restoremail -p nh -f nhusername.tgz -n 2 -u nhusername

• • • End of old instructions ***

• Backup locations *

backup-1: backup-3.nethere.net fpweb-1.nethere.net home-1.nethere.net koi.nethere.net marmaduke.inetworld.net ntdb-1.nethere.net ntweb-1.nethere.net ntweb-3.nethere.net ntweb-5.nethere.net ntweb-7.nethere.net phoenix.nethere.net unixweb-1.nethere.net unixweb-3.nethere.net unixweb-5.nethere.net unixweb-7.nethere.net shark.nethere.net tetra.nethere.net wms-1.nethere.net

backup-2: andromeda.nethere.net backup-4.nethere.net eel.nethere.net fpweb-2.nethere.net ntdb-2.nethere.net ntweb-2.nethere.net ntweb-4.nethere.net ntweb-6.nethere.net unixweb-2.nethere.net unixweb-4.nethere.net unixweb-6.nethere.net ds.znet.com mx1.znet.com mx2.znet.com mx3.znet.com dmx.znet.com la.znet.com uf.znet.com

backup-3: backup-1.nethere.net lists-1.nethere.net mailbox-1.mail.nethere.net mailbox-3.mail.nethere.net mta-1.mail.nethere.net mx-1.nethere.net nscache-1.nethere.net nsrbl-1.nethere.net pegasus.nethere.net scan-1.mail.nethere.net sndg-netapp-2.nethere.net relay-1.mail.nethere.net webmail-1.mail.nethere.net

backup-4: ahi.nethere.net backup-2.nethere.net cp-1.nethere.net cygnus.nethere.net dragon.nethere.net lisa.nethere.net mailbox-2.mail.nethere.net mta-2.mail.nethere.net news-1.nethere.net nscache-2.nethere.net nsrbl-2.nethere.net sawfish.nethere.net scan-2.mail.nethere.net scribe.nethere.net sndg-netapp-1.nethere.net relay-2.mail.nethere.net urchin.nethere.net

Mail servers

Organization

                                NetHere Mail Servers
                                ====================




         Customers              Inbound Mail            Outbound Mail
     /\          ||                 ||                    /\
     ||          ||                 || smtp               ||
     ||          ||                 \/                    ||
     ||          ||             ServerIron                ||
     ||          ||                 ||                    ||
     ||          ||                 ||                    ||
     ||          ||   smtp          \/                    ||
     ||          || =========>  mta-1    mta-2  ====>   relay-1  <====  unixweb-* servers
     ||          ||                 ||   /\             relay-2         ntweb servers
http ||     pop3 ||                 ||   ||
     ||     imap ||                 \/   +--------------+
     ||          ||             ServerIron              |
     ||          ||                 ||                  |
     ||          ||                 ||                  |
     ||          ||                 \/                  |
     ||          ||             nsrbl-1   nsrbl-2       |
     ||          ||         (spamhaus, dcc blacklist)   |
     ||          ||                 ||                  |
     ||          \/                 \/                  |
     \/         mailbox-1  <=== scan-1    scan-2        |
  webmail <===> mailbox-2    (sendmail/amavisd/sophos)  |
          imap  mailbox-3                               |
                 /\     /\                              |
                 ||     ||                              \/
             nfs ||     +--------------------------->  mta-db
                 ||                                    (LDAP)
                 \/
             sndg-netapp-1
             sndg-netapp-2
             sndg-netapp-3

Overview

mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail. mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.

nsrbl-1.nethere.net- Realtime blacklist check using dccd and spamhaus. The mta servers query the ip address of the mail server against the blacklist before transferring the email to the scanning servers.

nsrbl-2.nethere.net

scan-2.mail.nethere.net- Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers. scan-2.mail.nethere.net Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.

mailbox-1.mail.nethere.net- Front end server for the mail directories. Queries LDAP server for miscellaneous customer information including the directory the email should be stored in, passwords, spam sensitivity levels, etc. before delivering email to appropriate directory. mailbox-2.mail.nethere.net mailbox-3.mail.nethere.net

Mail is stored in maildir format:

/nfs/<nfs_number>/<platform>/u/s/username

sndg-netapp-1-new.nethere.net- Solaris system using zfs set up as a nfs mount on mailbox-1, -2, and -3. Email is stored on these platforms for all customers. sndg-netapp-2-new.nethere.net sndg-netapp-3-new.nethere.net

mx-1.nethere.net- spooling mail server

mailx-1.nethere.net- all outbound mails originating from a web server is sent out this email server. All web servers are set to use mailx.nethere.net which resolves (through the serveriron) to either mailx-1.nethere.net and mailx-2.nethere.net. Since we shut down mailx-2, all outbound mail from the web servers goes through mailx-1. This was to prevent all the customers email from being blacklisted as a result of a web site being compromised.

mta-db.mail.nethere.net- database server for the mta servers. All postfix databases are stored here.

webmail.nethere.net- Hosts the webmail program for webmail.nethere.net, webmail.simplyweb.net and webmail.znet.net.

Client mail settings

Inbound Mail Server:

POP3 pop3.nethere.net pop3.znet.net pop3.simplyweb.net

IMAP imap.nethere.net imap.znet.net imap.simplyweb.net

Outbound Mail server:

mail.nethere.net mail.znet.net mail.simplyweb.net

Webmail:

webmail.nethere.net webmail.znet.net webmail.simplyweb.net

Control Panel:

cp.nethere.net cp.znet.net cp.simplyweb.net

nsrbl Disk Quota Warning

In the event you get a nagios regarding space on the two nsrbl servers, it is most likely the result of the dccd databases getting too large. To clear out space, log into each nsrbl server and execute the following:

df -h

cd /var/dcc/libexec

./stop-dccd

cd /var/dcc

rm dcc_db*

sync ; sync

cd /var/dcc/libexec

./start-dccd

ps auxw | grep dccd

Update outbound quota for a single ip address

We restrict the total number of emails that a customer can send to 2000 email recepients per 24 hour period. This is tracked by the ip address of the computer that connects to the mta server when it sends an outbound emal. The ip address and count is stored in the postfix database server, mta-db.mail.nethere.net. To increase the quota for a single ip address, you can log into either mta-1, mta-2, or console direcly into the database server. To use one of the mta servers:

1.  Log onto mta-1.mail
2.  Switch to mysql
$mysql -h 10.0.0.69 -u policyd_outbound -p
username:  policyd_outbound
password:  p0stf1x!
3.  Review database information
mysql>show databases;
4.  Switch to the policyd table. 
mysql>use policyd;
5.  Review table header information
mysql>show tables;
mysql>describe throttle;
6.  Verify the ip address is above quota
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx';   #this is the ip address of the account you are increasing.
7.  Increase the maxium number of recepients the ip address can send to
mysql>update throttle SET _rcpt_max = _rcpt_max + 20000 where _from='24.249.205.66';
8.  Verify
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx';

Remove a blocked mail server

The message in maillog helps identify where it is blocked.

Relaying denied due to excessive spam                             admin-2:/dist/files/nhmta/nethere/conf/postfix/common/client_reject
Sender address rejected: Relaying denied due to Spam              admin-2:/dist/files/nhmta/nethere/conf/postfix/common/sender_reject
Relaying denied due to SPAM                                       policyd???
Client host rejected: ... listed at zen.spamhaus.org=127.0.0.2    nsrbl-1:/named/rbldns/zones/*.spamhaus.org
status=sent (250 2.7.1 Ok, discarded, id=95886-33 - SPAM)         user mail protection in provisioning????

Remove a mail server from policyd blacklist

The ip address of a mail server can be placed on a blacklist for several reasons, to high of rate of inbound email from a single ip address, incorrectly formatted helo, etc. In the event you need to remove a mail server's ip from the policyd blacklist, you can log into either mta server of connect directly to the policyd database server, mta-db.mail.nethere.net. To remove the ip address using a mta server:

1.  Log onto mta-1.mail
2.  Switch to mysql
$mysql -h 10.0.0.69 -u policyd_inbound -p policyd
username:  policyd_inbound
password:  p0stf1x!
3.  Display database information
mysql>show databases;
4.  Use policyd database
mysql>use policyd;
4.  Display table information
mysql>show tables;
mysql>describe blacklist;
5.  Verify ip address has been blacklisted
mysql>select * from blacklist where _blacklist='xxx.xxx.xxx.xxx'; 
6.  Delete the entry
mysql>delete from blacklist where _blacklist='xxx.xxx.xxx.xxx';
7.  Verify
mysql> select * from blacklist where _blacklist='xxx.xxx.xxx.xxx';

The suspected mail server may also be in the helo table.

mysql> describe helo ;
1.  Check the IP address
mysql> select * from helo where _host='xxx.xxx.xxx.xxx' ;
2.  Check the server helo name 
mysql> select * from helo where _helo like '%server%' ;
3.  Delete the entry
mysql> delete from helo where _host='xxx.xxx.xxx.xxx' ;
mysql> delete from helo where _helo like '%server%' ;

Remove a mail server from spamhaus blacklist

ssh to nsrbl-1.mail and nsrbl-2.mail

cd /named/rbldns/zones 

find which file the IP Address is in.

grep <IP Addr> *.spamhaus.org
co -l <xxx.spamhaus.org>
vi xxx.spamhaus.org
ci -u <xxx.spamhaus.org>

Add ip address to client_reject

The client_reject list is one of the ways we combat spam. If we receive spam compaints from a mail server or otherwise identitfy a mail server as a source of spam, we can add it to a client_reject list. An email sent from a mail server on the client_reject list is rejected with a notice saying "Relaying denied due to excessive spam". To add an ip address to the list, we use admin-2. Once the list is updated on admin-2, it is pushed out to mta-1 and mta-2. To add an ip address on admin-2:

Log into admin-2.nethere.net.
bash-3.00# cd /dist/files/nhmta/nethere/conf/postfix/common
bash-3.00# co -l client_reject
RCS/client_reject,v  -->  client_reject
bash-3.00# vi client_reject

-Add the ip address to the bottom of the list in the appropriate format.  It you want to reject any email from the mail server, you would use:
xxx.xxx.xxx      REJECT Relaying denied due to excessive spam
where xxx.xxx.xxx is the /24 subnet of the mail server the spam originated from.  Do not place a period after the last octet.  For example, to add a /16 subnet you would use:
xxx.xxx          REJECT Relaying denied due to excessive spam
The REJECT key word tells postfix to reject the email.  If you want to allow email from a /24 subnet, use the following format:
xxx.xxx.xxx      OK
This tells postfix to accept email from that subnet.

bash-3.00# ci -u client_reject
RCS/client_reject,v  <--  client_reject
new revision: 1.286; previous revision: 1.285
enter log message, terminated with single '.' or end of file:
>> .
done
bash-3.00# make
/nethere/software/nhmta/postfix-2.4.5/sbin/postmap -C . client_reject
mv client_reject.db maps/hash/client_reject.db
bash-3.00# cd /dist/rdist
bash-3.00# gmake nhmta-update
updating host mta-1.mail.nethere.net
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject
updating host mta-2.mail.nethere.net
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject
bash-3.00#

Free mysql disk space on mta-db.mail.nethere.net

In response to nagios alert "[Nagios] PROBLEM alert - mta-db.mail.nethere.net/Disk mysql is WARNING", to clear space within the mysql database:

1. ssh to mta-db.mail.nethere.net.

2. Log into mysql and execute the following:

[root@mta-db user]# mysql -uroot -p

Enter password: dB@dm1N!

mysql> show databases;

mysql> use policyd;

mysql> show tables;

mysql> describe helo;

mysql> describe throttle;

mysql> describe throttle_from_instance;

mysql> select * from throttle_from_instance limit 10;

mysql> select now();

mysql> select unix_timestamp(now());

mysql> select count(*) from throttle_from_instance where _expire > 1426534697;

mysql> select max(_expire) from throttle_from_instance;

mysql> truncate table throttle_from_instance;

mysql> select count(*) from throttle_from_instance;

mysql> select * from throttle_from_instance;

mysql> show tables;

mysql> optimize table throttle_from_instance;

mysql> show databases;

mysql> use policyd;

mysql> quit

[root@mta-db user]# df -h

Errors

Common webmail errors

Error: --cut-- Fatal error: Call to undefined function: applicatio€”¬p() in /www/webmail.nethere.net/htdocs/x/m/templates/message/navbar.inc on line 7 --cut--

Solution: Restart apache on webmail-1 (apachectl restart)

Error: --cut-- ERROR There was an error sending your message: unable to add recipient [webhosting@nethere.com]: Invalid response code received from server --cut--

Solution: More than likely DNS/domain issue, but check mta-1.nethere.net to be sure: --cut-- Aug 30 15:08:54 mta-1 nh/smtpd[50501]: NOQUEUE: reject: RCPT from webmail-1.mail.nethere.net[66.63.128.181]: 450 <orders@mulligrins.com>: Sender address rejected: Domain not found; from=<orders@mulligrins.com> to=<webhosting@nethere.com> proto=ESMTP helo=<webmail.nethere.net> --cut-- Problem: Webmail shows "1-5" messages but inbox appears empty. POP shows messages #1,2,3,4,5... could not be retrieved... server response error cannot open the message file it's gone.

Solution: Ownership issue

1) Use the provisioning tool to determine mailbox location:

 a) Click on email account, Engineering
 b) Location is the "Storage Path:" 

2) Log in to either mail server then change ownership of maildir to mailuser:mailuser

 # chown -R mailuser:mailuser /nfs/#/platform/u/s/username

DEPRECIATED INFORMATION

Webservers

Allowing SSI (Server Side Includes)

• • • • • NOTE: This has been DEPRECIATED, all servers now allow SSI by default *****

We do not allow EXEC permissions for SSI.

1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts>

  # co -l www.domain.tld(.common)  

2) Add the following lines under the <Directory> directive --cut-- AddType text/html .shtml AddHandler server-parsed .shtml --cut--

3) Edit the "Options" line to be the following: --cut-- Options Indexes FollowSymLinks IncludesNOEXEC --cut--

4) Add a DirectoryIndex line above the <Directory> directive with the following: --cut-- DirectoryIndex index.shtml index.html index.htm home.html home.shtml index.php --cut--

• • In summary, the config should look something similar to this **

--cut-- DocumentRoot "/www/www.domain.tld/htdocs"

 DirectoryIndex index.shtml index.html index.htm home.html home.shtml 
 <Directory "/www/www.domain.tld/htdocs">
   AddType text/html .shtml
   AddHandler server-parsed .shtml
   Options Indexes FollowSymLinks IncludesNOEXEC
   Order allow,deny
   Allow from all
 </Directory>

--cut--

5) Check in Apache config

  # ci -u www.domain.tld(.common) < /dev/null  

6) Restart Apache

  # apachectl configtest
  # apachectl restart

More SSI notes can be found here: http://httpd.apache.org/docs/1.3/howto/ssi.html

SimplyInternet

Z-net

Migrating to LAMP Host Servers

Below are outlines of the steps to migrate website & email hosting from nethere to lamphost configured servers. This is almost certainly incomplete at this stage. The exact steps to follow can vary depending on specifics of the client, so use this as a guide only. Be ready to adjust commands or add steps as needed.

Edit Nethere DNS

ssh user@admin-1.nethere.net

sudo -i

ssh ns1.nethere.net

cd /named/master

co -l example.com

vi example.com

ci -u example.com

rndc reload example.com

Make sure to increment serial number by 1 each time a change is made.

Migrate Website to nh5.jcihosting.net

Set TTL to 300 on ns1.nethere.net

Open up these web admins to get current account/site info:

http://toolbox.nethere.net/ -> Provisioning Tool (search for domain name)

https://secure.johncompanies.com/mgmt/index.html (search for col#####)

1. Create the lamphost user.

https://www.lamphost.com/admin/user/user/create

username: col#####

email: same as listed for col##### account

password: random, secure, won't be shared with client

defaults on rest of page

2. Create the customer account

https://nh5.jcihosting.net:8443/

LAMP Host Users -> Add User -> col#####

Virtual Hosts -> Add Virtual Host

Username: Select same username as above from pulldown

Domain: example.com

Hosting Plan: select plan that matches what's listed in https://secure.johncompanies.com/mgmt/view.html?cid=col#####

Add default DNS: No

3. Virtual Hosts -> Site Manager (next to domain added)

Site Manager Users -> Add User

Username: can be anything - I typically use Optigold Login from nethere admin

Password: make up something secure and record - this one will go to the user

4. Email & FTP -> Add Account

Email Account: ftp

Password: make up something secure and record - this one will go to the user

Enable FTP Access: Yes

5. Optional: Add MySQL database if site needs it (WordPress etc)

ssh to nh5.jcihosting.net and run 'add_mysql.php <example.com>'

MySQL database info will be emailed to webmaster@lamphost.com (and output on command line - ignore instructions to run additional commands)

https://mail.lamphost.com Sm77DdnQ

6. Copy over website

Login via ssh nh5

ssh YOURLOGIN@nat.johncompanies.com ssh root@nh5.jcihosting.net

cd /var/www/example.com/

rsync -v --archive --one-file-system --delete --delete-during --rsh=/usr/bin/ssh dsmith@unixweb-10.nethere.net:/www/www.example.com/htdocs ./

Change 'matt' to your username. Note that I had to add my username to groups httpd & webuser to allow me to copy website files without being root.

chown -R col#####:col##### htdocs

7. If site has MySQL data find the connection info. For example, WP sites will have the info in htdocs/wp-config.php

mysqldump -h mysqldb-1.webhost.nethere.net -u username --password=password databasename > databse.sql

mysql exampledotcom < database.sql

rm database.sql

8. If site is ready to be made live on nh5.lamphost.net then update DNS on ns1.nethere.com

Send email to client with details. Search support@jcihosting.com Sent folder for emails with Subject "updated hosting for" for example emails. You will have to adjust as needed (username/passwords, different info based on specific client).

9. After you finish, update the johncompanies.com backend for the col0# CID and change the nethere hosting server to nh3.jcihosting.net.

10. To add a SSL certificate

       letsencrypt-create.php <domain.com>

   To remove a SSL certificate
       del-letsencrypt-certificate.php <domain.com>

Migrate Email to nh5.jcihosting.net

If migrating website & emails both follow the above steps to transfer the website first.

If migrating emails only the follow the first steps 1-3 of the above steps to set up the virtualhost on nh3.

1. Set TTL for MX record to 300.

2. Generate email report:

Log in to http://toolbox.nethere.net/

Select Provisioning Tool and search for the domain

Navigate to the Email section and then Generate Report.

Select these fields for the report:

Mail address

Alias

Forwarding address

Mail storage path

Copy the result of the report and save it as a text file in /tmp/ on nh5.jcihosting.net named example.com.txt (replace domain name)

3. Create the e-mail accounts

Make sure the virtualhost has enough email accounts allocated before proceeding. Edit the virtualhosts.virtualhost_settings table directly if need be to increase allocation.

   Log in here: https://nh5.jcihosting.net:8443/
   Select "MySQL.LH"
   Go to the "virtualhosts" database
   Search the "virtualhost_settings" settings for 'example.com' 
      and adjust the 'email_accounts' setting.

Run this script once and only once:

/usr/local/bin/migrate_mail_accounts_from_nethere.php example.com

That will create all email accounts and aliases.

A file containing the list of email accounts and their new passwords will be created as /tmp/example.com-passwords.txt. Grab a copy of that for sending to the client.

4. Sync the emails

Another file will be created that contains the bash commands to sync emails from the old server. This will be named /tmp/example.com-sync.sh

Copy the file somewhere like the /root/ home dir and add "#!/bin/bash" as the first line and change perms to 750. Now you can run this script to sync all mails from nethere to nh3. Run it immediately to get an initial sync of emails.

When ready to switch email hosting over run the sync script one more time, then update DNS on nh3.nethere.net.

If you'd like to sync again after the move (if there was a delay or you think there may have been incoming emails during the transtion) remove the '--del' option from the sync script and then re-run.

5. Send email to customer

Search support@jcihosting.com Sent folder for messages with Subject "new email hosting for" for examples of emails to use for communicating with the client.

6. Update DNS and put NetHere mail on hold Once DNS has been pointed to the new server, go into the NetHere Toolbox http://toolbox.nethere.net/ and go to Domain -> General and then set Status to 'On Hold'.

7. Update host in JohnCompanies database.

Moving between Host Servers on Lamphost

You should delete the old virtual host before creating the new virtual host. It may take up to an hour to delete the old virtual host. To speed up the delete:

  /opt/lamphost/cron-bin/del_virtualhosts.php

If you create the new virtual host before, you will need to update the mylampsite.com:

  /opt/lamphost/bin/update_mylampsite_subdomain.php databaseproviders.com

Upgrade a Lamphost server to increase database limit

I've increased the database limit for this customer to 2. Now they can add a second db via the Site Manager admin.

Here is how I made the adjustment (we don't have a friendly admin for this function):
Log in to Server Manager: https://nh3.jcihosting.net:8443/
Select the "MySQL.LH" phpMyAdmin
Select 'virtualhosts' db.
Adjust the 'databases' setting for the argee.com domain in the 'virtualhost_settings' table.

looking in the mail log files

exigrep