BigBrother

Overview

BB is our monitoring service. It runs on mail and does 2 things:

1. It watches machines (ping and ssh)
2. It listens to bb clients running on other machines for reports of problems on those machines (CPU, disk, procs)

When things go wrong, there are 2 status levels, warn and panic. Panic events will always page. Warn events will only show up on the bb status page. The link to the status page can be found under Mgm. -> BigBrother (https://secure.johncompanies.com/mgmt/bb/)

Condition purple means it is no longer receiving updates from a particular machine. You will still get this even if you've removed the host from bb-hosts. To remove fully, you have to run the bbrm <hostname> command to remove all prev logs.

NOTE: may not be working...If you get a page from bb about some alarm situation, you can silence that alarm by responding with (in the subject line): Re: !BB - <7 digit unique alarm id send in page>! ACK=Y DELAY=<# mins you want it silenced>

Or, you can go to the bb status page and clicking on the lightning bolt at the top: https://secure.johncompanies.com/mgmt/bb/help/bb-ack.html

Changing who get’s paged

mail# su bb
%cd
%cd bbsrc/bb1.9i-btf/etc
%vi bbwarnrules.cfg
(edit entries)
%cd ..
%./runbb.sh restart
Stopping Big Brother...
        Starting Big Brother Daemon (bbd)...
        Starting Network tests (bb-network)...
        Starting Display process (bb-display)...
Big Brother 1.9i started
%exit

Shutting down bb

mail# su bb
%cd
%cd bbsrc/bb1.9i-btf/etc
%vi bbwarnrules.cfg
(edit entries)
%cd ..
%./runbb.sh stop
%exit

Disabling monitoring for a server

If there’s ever a situation where a machine is causing too many pages, you can stop it’s monitoring as follows:

mail# su bb
%cd
%cd bbsrc/bb1.9i-btf/etc
%vi bb-hosts	
(comment out entry, i.e.:)

#10.1.4.61 virt11.johncompanies.com # ssh

%cd ..
%./runbb.sh restart
Stopping Big Brother...
        Starting Big Brother Daemon (bbd)...
        Starting Network tests (bb-network)...
        Starting Display process (bb-display)...
Big Brother 1.9i started
%exit

monitor.johncompanies.com

This is the monitoring service we offer to customers (colo) that have asked for monitoring. It's running under VEID 5 on quar1. It has been altered so when customers are paged (emailed), the url they are given takes them to a status page only they can see. The general/overview status page is password protected (password needs to be reset, it's lost).

Compiling from scratch

jail9 /home/bb/bbc1.9i-btf/install# ./bbconfig

        Welcome to the Big Brother installation procedure.
Please confirm your OS: [freebsd]:
---> We'll configure for freebsd...

        Big Brother needs it's own user id.  If your BB user doesn't exist,
        you'll have to create it then re-run bbconfig.

What will be the user ID for BB [bb]:
---> BB will only run from user 'bb'

        Making sure BBHOME </usr/home/bb/bbc1.9i-btf> is writable...

---> OK, /usr/home/bb/bbc1.9i-btf is fine...


        When you set up your machines, you should use Fully Qualified
        Domain names, this means you use the whole name, like www.bb4.com,
        instead of just 'www'.  This is recommended.

Use FQDN (y/n): [y] y
---> Good, we'll use FQDN

--------------------------------------------------------

--> UPDATING runbb.sh
--> UPDATING bbsys.local
--> CHECKING COMMAND PATHNAMES
*** Verifying pathnames to necessary commands...
*** The following changes need to be made...
        --> /usr/sbin/nslookup changed to /usr/bin/nslookup
*** Making changes...

*** We've noticed that we've set some of the pathnames wrong
*** by default here.  May we mail a summary of the paths we
*** missed back to info@bb4.com so we can update our installs?
[y/n]
n
*** Done.
--> UPDATING bbdef.sh

--------------------------------------------------------


--------------------------------------------------------

--> Done.  Now do
                chown -R bb /usr/home/bb/bbc1.9i-btf

Now compile Big Brother

                cd /usr/home/bb/bbc1.9i-btf/src
                make
                make install

--------------------------------------------------------

jail9 /home/bb/bbc1.9i-btf/install#
jail9 /home/bb/bbc1.9i-btf/install# cd /usr/home/bb/bbc1.9i-btf/src
jail9 /usr/home/bb/bbc1.9i-btf/src# make
jail9 /usr/home/bb/bbc1.9i-btf/src# make install

copy over the following from an older install's /etc dir:
bbdef-client.sh
bbdef.sh
bb-hosts
bb-bbexttab
bb-dftab
bb-msgstab

# for f in bbdef-client.sh bbdef.sh bb-hosts bb-bbexttab bb-dftab bb-msgstab; do cp bbc1.9e-btf/etc/$f bbc1.9i-btf/etc/;done

cp bbc1.9e-btf/ext/openfiles bbc1.9i-btf/ext/openfiles

jail9 /home/bb# chown -R bb bbc1.9i-btf/